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United  States  General  Accounting  Office 
Washington,  D.C.  20548 


February  28,  2001 

The  Honorable  Walter  W.  Herger 
Chairman,  Subcommittee  on  Human  Resources 
Committee  on  Ways  and  Means 
House  of  Representatives 

The  Honorable  Nancy  L.  Johnson 
House  of  Representatives 

Over  4  years  have  passed  since  federal  welfare  reform  legislation  was 
enacted,  replacing  the  legal  entitlement  to  cash  assistance  under  the  Aid  to 
Families  With  Dependent  Children  (AFDC)  program  with  the  promotion  of 
work  and  personal  responsibility.1  The  policy  and  research  community  has 
been  asking  what  has  happened  to  low-income  families  since  the  Personal 
Responsibility  and  Work  Opportunity  Reconciliation  Act  of  1996 
(PRWORA)  created  the  Temporary  Assistance  for  Needy  Families  (TANF) 
block  grant,  replacing  the  federal  entitlement  to  assistance  and 
institutionalizing  reforms  that  had  been  initiated  by  many  states.  Overseen 
by  the  Department  of  Health  and  Human  Services  (HHS)  at  the  federal 
level,  TANF  makes  $16.8  billion  in  federal  funds  available  to  states  each 
year  through  2002.  While  TANF  delegates  wide  discretion  to  the  states  to 
design  and  implement  the  program,  it  does  specify  four  broad  program 
goals  that  focus  on  children  and  families: 

•  providing  assistance  to  needy  families  so  that  children  may  be  cared  for 
in  their  own  homes  or  in  the  homes  of  relatives; 

•  ending  the  dependence  of  needy  parents  on  government  benefits  by 
promoting  job  preparation,  work,  and  marriage; 

•  preventing  and  reducing  the  incidence  of  out-of-wedlock  pregnancies; 
and 

•  encouraging  the  formation  and  maintenance  of  two-parent  families. 

As  TANF  has  been  implemented  by  the  states,  the  Congress  has  asked 
questions  about  a  broad  range  of  issues  concerning  welfare  reform’s 
consequences.  Several  issues — for  example,  the  circumstances  of  those 
who  have  left  TANF  and  the  characteristics  of  those  currently  receiving 
cash  assistance — relate  to  what  has  happened  to  low-income  families  since 


lrritle  I  of  the  Personal  Responsibility  and  Work  Opportunity  Reconciliation  Act  of  1996,  P.L. 
104-193,  established  the  TANF  block  grant  program. 
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TANF  has  been  in  effect.  The  Congress  also  has  asked  about  the  extent  to 
which  any  changes  that  have  occurred  in  low-income  families’  life 
circumstances,  such  as  TANF  recipients’  increases  in  employment  and 
earned  income  and  decreases  in  births,  can  be  attributed  to  TANF  rather 
than  to  other  factors,  such  as  changes  in  economic  conditions.  These 
questions  will  be  especially  important  when  the  program  is  considered  for 
reauthorization  in  2001. 

To  determine  what  data  will  be  available  from  planned  and  existing 
research  to  address  these  and  other  questions,  you  asked  us  to  examine  (1) 
the  type  of  data  that  will  be  available  from  national  surveys  and  studies  of 
welfare  reform  to  help  assess  progress  towards  TANF’s  goals  and  (2)  the 
usefulness  of  the  data  for  assessing  TANF’s  progress  nationwide.  Our  major 
objective  was  to  examine  data  already  collected  by  national  surveys  of  low- 
income  families  and  for  major  studies  of  welfare  reform  to  identify  data 
available  for  use  in  a  comprehensive  assessment  of  TANF  for  the 
reauthorization  debate.  We  did  not  examine  study  findings;  rather,  we 
examined  the  data  that  have  been  collected  or  will  be  collected  as  part  of 
the  considerable  body  of  research  about  TANF  that  is  accumulating.2  The 
focus  on  data  also  addresses  issues  raised  by  several  critiques  of  this  work 
that  have  emerged.  The  critiques  that  examine  the  methodology  of  this 
research  agree  that,  for  welfare  reform  research,  the  quality  and 
comprehensiveness  of  the  existing  data  are  cause  for  concern.  Two 
critiques  in  particular  question  whether  the  major  national  survey  designed 
to  provide  information  to  assess  TANF  at  the  national  level — the  Survey  of 
Program  Dynamics  (SPD) — has  “a  sufficient  number  of  observations”  for 
an  analysis  of  TANF’s  effects  in  individual  states.3 

To  determine  what  data  will  be  available,  we  considered  all  of  the  national 
surveys  and  welfare  reform  studies  discussed  in  six  major  critiques  of 


2See  Christine  Devere,  Gene  Falk,  and  Vee  Burke,  Welfare  Reform  Research:  What  Have  We 
Learned  Since  the  Family  Support  Act  of 1988?  (Washington,  D.C.:  Congressional  Research 
Service,  Oct.  20,  2000)  for  a  summary  of  findings  about  the  effects  of  welfare  reform 
initiatives. 

;>The  Census  Bureau  conducts  the  SPD,  a  longitudinal  survey  of  a  nationally  representative 
sample  of  families,  with  emphasis  on  eligibility  for  and  participation  in  welfare  programs, 
employment,  earnings,  the  incidence  of  out-of-wedlock  births,  and  family  well-being.  The 
provision  in  PRWORA  that  authorized  the  SPD  specified  that  the  sample  be  drawn  from  the 
1992  and  1993  Survey  of  Income  and  Program  Participation  (SIPP)  samples.  However,  the 
original  samples  for  the  SIPP  were  designed  to  be  nationally  representative,  not  to  be 
representative  of  any  state. 
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welfare  reform  research,  as  well  as  surveys  and  studies  available  through 
the  Research  Forum  and  the  Welfare  Information  Network,  the  major 
clearinghouses  for  welfare  reform  research.4  We  also  interviewed  HHS 
officials  about  their  perspective  on  welfare  reform  data  and  asked  welfare 
experts  to  nominate  key  pieces  of  work  in  the  field  for  inclusion  in  our 
review.  We  selected  a  sample  of  surveys  and  studies  that  resulted  in  187 
individual  data  sets  related  to  TANF  reauthorization  issues.5  For  each  data 
set,  we  analyzed  the  content  of  the  data  to  determine  (1)  the  range  of 
TANF-related  data  topics  covered;  (2)  the  level  of  detail  the  data  offer;  (3) 
the  data’s  quality,  including,  where  relevant,  the  sampling  method,  sample 
size,  response  rate,  and  attrition  rate;  and  (4)  how  the  data  could  be  used  to 
provide  the  Congress  with  an  understanding  of  what  has  happened  under 
TANF.  Appendix  I  provides  additional  details  about  our  methodology  and 
its  limitations.  We  conducted  our  work  between  January  2000  and  January 
2001  in  accordance  with  generally  accepted  government  auditing 
standards. 


Results  in  Brief  National,  state,  and  local  data  will  be  available  to  help  assess  progress 

toward  TANF’s  goals,  but  these  data  address  the  goals  to  differing  degrees. 
National  data,  which  include  data  collected  in  national  surveys  and 
information  that  all  states  report  to  HHS,  include  extensive  information 
related  to  TANF’s  two  goals  of  providing  assistance  to  needy  families  and 
ending  dependency  on  government  benefits  but  have  limited  information 
about  TANF’s  goals  of  preventing  out-of-wedlock  pregnancies  and 
promoting  family  formation.  The  data  pertain  to  such  issues  as  changes  in 
the  TANF  caseload,  use  of  noncash  assistance  by  current  and  former  TANF 
recipients,  recipients’  participation  in  work  activities,  employment  status 
and  earnings,  and  family  well-being.  Although  there  are  national  data  on  the 
incidence  of  out-of-wedlock  births  and  marriage  among  TANF  recipients 
and  other  low-income  families,  these  data  include  only  very  recently 
available  information  on  states’  strategies  to  prevent  out-of-wedlock 
pregnancies  or  promote  family  formation.  Data  from  studies  of  welfare 
reform  at  the  state  and  local  levels  contain  the  same  kind  of  information  as 
national  data,  but  they  also  include  information  about  areas  very  recently 


4See  fig.  8  in  app.  I  for  a  list  of  the  major  critiques. 

bA  data  set  is  a  body  of  data  gathered  for  analysis  by  a  single  data  collection  method,  such  as 
by  survey  or  from  welfare  case  records  or  the  administrative  records  of  programs  such  as 
the  Unemployment  Insurance  or  federal  income  tax  systems. 
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covered  by  national  data.  For  example,  data  are  available  about  the 
birthrates  and  marriage  patterns  of  individuals  who  participated  in 
programs,  some  of  which  had  provisions  intended  to  foster  marriage  and 
discourage  out-of-wedlock  pregnancy.  Much  of  these  data  come  from 
waiver  evaluations — evaluations  conducted  in  states  that  experimented 
with  their  welfare  program,  under  a  waiver  from  HHS,  prior  to  TANF. 

The  usefulness  of  existing  data  for  assessing  TANF’s  progress  varies. 
National  data  can  be  analyzed  to  develop  a  descriptive  picture  of  TANF  for 
the  nation  as  a  whole.  However,  of  the  seven  national  data  sets  we 
reviewed,  only  two  can  be  used  to  describe  the  lives  of  families  receiving 
TANF  within  individual  states.  To  gain  information  about  TANF’s  effects  in 
the  states,  data  from  waiver  evaluations  and  demonstrations  can  be  used 
instead.  In  many  cases,  these  data  were  collected  by  randomly  assigning 
welfare  recipients  to  groups  subject  to  different  welfare  provisions, 
allowing  employment,  earnings,  and  well-being  of  the  groups  to  be 
attributed  to  the  provisions.  However,  most  waiver  and  demonstration  data 
were  collected  in  localities  and  so  cannot  be  used  to  assess  state-level 
effects.  We  examined  nearly  40  data  sets  that  could  be  analyzed  for 
information  about  the  circumstances  of  former  recipients.  However,  only  a 
subgroup  of  these  data  sets  met  criteria  that  allowed  the  sample  to  be 
generalized  statewide.  These  data  sets  represented  15  states.  In  some 
cases,  the  value  of  survey  data  collected  from  those  who  left  welfare  was 
limited  because  few  former  recipients  actually  responded  to  the  surveys.  In 
general,  the  need  for  information  about  TANF’s  progress  will  have  to  be 
balanced  against  the  challenges  of  rigorous  data  collection  from  the  low- 
income  population. 


Background 


PRWORA  overhauled  the  nation’s  welfare  system  by  abolishing  the 
previous  welfare  program,  AFDC,  and  creating  the  TANF  block  grant. 
PRWORA  established  four  broad  goals  for  TANF,  which  included  (1) 
providing  assistance  to  needy  families  so  that  children  may  be  cared  for  in 
their  own  homes  or  in  the  homes  of  relatives;  (2)  ending  dependence  of 
needy  parents  on  government  benefits  by  promoting  job  preparation,  work, 
and  marriage;  (3)  preventing  and  reducing  the  incidence  of  out-of-wedlock 
pregnancies;  and  (4)  encouraging  the  formation  and  maintenance  of  two- 
parent  families.  Unlike  the  previous  program,  TANF  gives  states  great 
flexibility  to  design  programs  that  meet  these  goals.  However,  while  states 
have  flexibility,  the  programs  they  design  must  meet  several  federal 
requirements  that  emphasize  the  importance  of  work  and  the  temporary 
nature  of  TANF.  For  example,  PRWORA  requires  that  parents  receiving 
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assistance  engage  in  work,  as  defined  by  the  state,  after  receiving 
assistance  for  24  months,  or  earlier,  at  state  option.  In  exercising  their 
option,  28  states  require  immediate  participation  in  work,  and  9  other 
states  require  participation  in  work  within  6  months  of  receiving  cash 
assistance,  resulting  in  great  interstate  variation  in  program  provisions. 
Further,  despite  the  programmatic  flexibility  authorized  by  TANF,  states 
must  meet  federal  data  reporting  requirements  by  submitting  quarterly 
reports  that  include  information  from  administrative  records  about  those 
receiving  welfare  and  those  terminated  from  assistance,  as  well  as  an 
annual  report,  to  HHS.  The  annual  report  contains  information  about 
program  characteristics,  such  as  states’  activities  used  to  prevent  out-of- 
wedlock  pregnancy. 

In  1995,  we  reported6  that  the  block  grants  enacted  as  part  of  the  Omnibus 
Budget  Reconciliation  Act  of  1981  (OBRA)  carried  no  uniform  federal 
information  requirements. '  We  found  that  the  program  information  states 
collected  was  designed  to  meet  individual  states’  needs  and  that,  as  a 
result,  it  was  difficult  to  aggregate  states’  experiences  and  speak  from  a 
national  perspective  on  the  block  grant  activities  or  their  effects.  Without 
uniform  information  definitions  and  collection  methodologies,  it  was 
difficult  for  the  Congress  to  compare  state  efforts  or  draw  meaningful 
conclusions  about  the  relative  effectiveness  of  different  strategies.  In  a 
second  examination  of  federal  block  grant  programs,  we  reported  that 
problems  in  information  and  reporting  under  many  block  grants — the 
Education  Block  Grant,  the  Community  Services  Block  Grant,  and  the 
Alcohol,  Drug  Abuse,  and  Mental  Health  Services  Block  Grant — have 
limited  the  Congress’  ability  to  evaluate  them.8  However,  for  the  TANF 
Block  Grant,  the  regulations  require  that  states  submit  the  quarterly  TANF 
Data  Report  and  the  TANF  Financial  Report  or  be  subject  to  statutory 
penalties.  For  these  reports,  HHS  provides  data  reporting  specifications 
including  timing,  format,  and  definitions  for  such  data  topics  as  family 
composition,  employment  status,  and  earned  and  unearned  income.  These 


6 Block  Grants:  Issues  in  Designing  Accountability  Provisions  (GAO/AIMD-95-226,  Sept.  1, 
1995). 

'OBRA  created  nine  block  grants:  Community  Services;  Alcohol,  Drug  Abuse,  and  Mental 
Health  Services;  Primary  Care;  Social  Services;  Maternal  and  Child  Health;  Preventive 
Health  and  Health  Services;  Education  (ch.  2);  Low-Income  Home  Energy  Assistance;  and 
Community  Development  (small  cities). 

H  Block  Grants:  Characteristics,  Experience,  and  Lessons  Learned  (GAO/HEHS-95-74,  Feb.  9, 
1995). 
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specifications  facilitate  the  use  of  HHS’  TANF  administrative  data  for 
welfare  reform  research  by  improving  the  data’s  comparability  from  state 
to  state.  Several  national  surveys  and  data  collected  for  state  and  local 
studies  of  welfare  reform  also  are  potential  sources  of  data  for  an 
assessment  of  TANF. 


National  Data  Sources  on  A  number  of  national  surveys  that  collect  information  about  welfare  receipt 
TANF  have  been  used  in  the  past  by  researchers  to  analyze  welfare  reform  or 

have  been  developed  to  assess  current  welfare  reform.  Four  surveys — the 
Survey  of  Income  and  Program  Participation  (SIPP),  the  Current 
Population  Survey  (CPS),  the  National  Longitudinal  Survey  of  Youth 
(NLSY),  and  the  Panel  Study  of  Income  Dynamics  (PSID) — have  been  used 
in  past  research  on  the  AFDC  program  and  the  low-income  population  in 
general.  Both  the  SIPP  and  the  PSID  have  updated  their  questionnaires  to 
include  questions  that  pertain  to  welfare  reform  specifically,  including 
questions  about  the  work  participation  requirements  and  penalties  for  not 
complying  with  these  and  other  program  rules.  Moreover,  two  national 
surveys  are  designed  specifically  to  answer  questions  about  welfare 
reform.  The  U.S.  Census  Bureau,  at  the  direction  of  the  Congress,  is 
conducting  a  longitudinal  survey  of  a  nationally  representative  sample  of 
families,  with  emphasis  on  eligibility  for  and  participation  in  welfare 
programs,  employment,  earnings,  the  incidence  of  out-of-wedlock  births, 
and  adult  and  child  well-being.  This  survey,  the  Survey  of  Program 
Dynamics,  was  designed  to  help  researchers  understand  the  impact  of 
welfare  reform  on  the  well-being  of  low-income  families  and  children. 
Similarly,  the  Urban  Institute  has  been  conducting  a  multiyear  project 
monitoring  program  changes  and  fiscal  developments,  along  with  changes 
in  the  well-being  of  children  and  families.  Part  of  this  project  includes  a 
nationally  representative  survey  of  50,000  people  called  the  National 
Survey  of  America’s  Families  (NSAF)  that  is  collecting  information  on  the 
well-being  of  adults  and  children  as  welfare  reform  is  implemented.9 

With  the  change  in  the  fundamental  structure  of  the  nation’s  welfare 
program,  there  have  been  several  efforts  by  private  research  organizations 
to  document  the  policies  states  have  adopted  under  TANF.  The  Center  for 
Law  and  Social  Policy  and  the  Center  on  Budget  and  Policy  Priorities,  in 
collaboration,  have  created  the  State  Policy  Documentation  Project  to 


’’Although  valuable  resources  for  assessing  national  programs,  national  surveys  often  are 
costly.  For  example,  the  1997  NSAF  cost  $14  million  to  administer  and  process. 
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document  policies  in  all  50  states  and  the  District  of  Columbia.  Available  on 
the  Web,  the  State  Policy  Documentation  Project  contains  information 
about  state  policies  contained  in  statutes,  regulations,  and  caseworker 
manuals,  but  it  does  not  describe  state  practices.  In  addition,  the  Urban 
Institute  has  developed  and  made  available  to  the  public  a  database  that 
documents  changes  in  state  program  rules  since  1996. 


State  and  Local  Data  From  Prior  to  and  since  TANF’s  implementation,  a  considerable  body  of  research 

Welfare  Reform  Research  about  the  low-income  population  has  been  conducted  to  examine  the 

circumstances  of  families  affected  by  welfare  reform,  the  effectiveness  of 
welfare  reform  initiatives,  and  the  implementation  of  TANF  at  the  state 
level.  HHS  has  played  a  major  role  in  laying  the  foundation  for  this  welfare 
reform  research.10  During  the  early  1990s,  HHS  granted  waivers  to  states 
that  allowed  them  to  test  various  welfare  reform  provisions.  In  return, 
states  were  required  to  evaluate  the  effectiveness  of  the  waiver  provisions 
by  randomly  assigning  welfare  recipients  to  either  participate  in  the  waiver 
program  or  not.  With  the  passage  of  TANF,  states  were  given  the  option  to 
continue  their  waiver  evaluations  as  originally  designed  or  modify  the 
evaluation  design.  Several  states  opted  to  continue  with  their  original 
random  assignment  design,  while  others  modified  their  evaluation  designs 
to  focus  on  examining  the  implementation  of  the  waivers  or  describe 
participants’  employment,  earnings,  and  well-being.  Because  some 
elements  of  the  waivers  granted  to  states  were  incorporated  into  many 
TANF  programs,  the  waiver  evaluations  provide  useful  insights  into  issues 
and  designs  for  research  about  TANF. 

However,  according  to  HHS,  one  aspect  of  waiver  policies  may  mean  that 
some  waiver  evaluations  may  not  represent  TANF  requirements 
completely.  TANF  established  work  requirements  for  all  adult  recipients, 
but  states  could  delay  adhering  to  these  requirements  under  their  TANF 
program,  in  part  or  whole,  if  the  requirements  were  inconsistent  with  state 
waiver  policies.  Under  the  Job  Opportunities  and  Basic  Skills  Training 
(JOBS)  program,  work  requirements  were  mandatory  for  a  work-ready  or 
able-bodied  population,  excluding  a  number  of  subgroups  such  as  those 
caring  for  young  children  and  the  disabled.  For  the  most  part,  states  that 
continued  the  original  random  assignment  design  maintained  some  or  all  of 


10The  Congress  also  has  supported  the  use  of  rigorous  evaluations.  For  example,  in  1988,  the 
Family  Support  Act  authorized  fimding  for  research  using  random  assignment  that  led  to  a 
major  examination  of  welfare-to-work  strategies  that  existed  before  TANF. 
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the  JOBS  exemptions  from  work  requirements  and  applied  these 
exemptions  in  determining  who  was  subject  to  time-limited  assistance. 
Consequently,  while  these  states’  waivers  may  incorporate  other  work 
policies  prescribed  under  TANF,  these  policies  would  not  be  expected  to 
affect  the  exempt  population.  In  contrast,  in  states  that  do  not  claim  JOBS 
exemptions  from  work  requirements,  all  adults  are  subject  to  work 
requirements  and  time  limits  on  assistance.  Thus,  while  testing  TANF-like 
policies,  evaluations  that  continued  the  random  assignment  design  may  not 
fully  reflect  the  experience,  outcomes,  or  impacts  of  fully  implemented 
TANF  requirements. 

In  addition  to  the  waiver  evaluations,  HHS,  as  well  as  private  foundations, 
has  provided  funding  for  demonstration  programs  across  the  country.  The 
demonstration  programs  are  pilot  projects  designed  to  measure  the  effects 
of  a  particular  strategy,  rather  than  an  entire  program,  on  welfare  recipients 
or  those  eligible  to  receive  welfare.  Many  of  these  demonstration  programs 
were  intended  to  increase  employment,  decrease  out-of-wedlock 
pregnancy,  or  promote  marriage.  For  example,  in  the  late  1980s,  several 
demonstration  programs  aimed  at  decreasing  teen  pregnancy  among 
welfare  recipients  were  developed.  One  program,  the  New  Chance 
Demonstration,  randomly  assigned  teen  mothers  receiving  welfare  to 
participate  in  a  program  that  offered  education  or  training  classes  and 
other  support  services  and  then  compared  the  accomplishments  of  these 
teen  mothers  with  those  of  teen  mothers  who  did  not  participate  in  the 
program. 

Given  states’  greater  responsibility  for  welfare  programs  under  PRWORA 
and  the  larger  number  of  people  leaving  the  welfare  rolls,  there  has  been 
general  interest  among  program  administrators  and  state  and  local 
policymakers  about  the  condition  of  those  who  are  no  longer  receiving 
TANF,  otherwise  known  as  “leavers.”  In  response  to  this  concern,  a 
growing  body  of  research  about  leavers  has  been  initiated  at  both  the  state 
and  federal  levels.  Generally,  researchers  have  found  that  once  low-income 
families  leave  welfare,  they  become  hard  to  keep  track  of.  Moreover,  we 
previously  reported  that  studies  of  former  TANF  recipients’  status  differ  in 
important  ways,  including  geographic  scope,  the  time  period  covered,  and 
the  categories  of  families  studied,  which  limits  the  comparability  of  the 
data  across  states.11  In  order  to  facilitate  cross-state  study  comparisons,  the 


nSee  Welfare  Reform:  Information  on  Former  Recipients’  Status  (GAO/HEHS-99-48,  Apr.  28, 
1999). 
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Office  of  the  Assistant  Secretary  for  Planning  and  Evaluation  (ASPE) 
within  HHS  has  issued  guidance  to  states  and  the  research  community  on 
developing  comparable  measures  for  commonly  reported  outcomes  and 
defined  these  outcomes.  In  fiscal  year  1998,  ASPE  awarded  approximately 
$2.9  million  in  grants  to  10  states  and  three  large  counties  to  study  leavers, 
followed  by  additional  grants  in  fiscal  years  1999  and  2000.  ASPE  also  has 
encouraged  the  researchers  to  use  comparable  measures. 

Research  is  also  being  conducted  to  examine  the  effects  of  welfare  reform 
in  metropolitan  areas  or  neighborhoods.  This  area  of  research  is  important 
because  the  caseload  decline  in  urban  areas  has  been  substantially  lower 
than  in  other  areas  of  the  country.  Moreover,  urban  areas  can  have  higher 
unemployment  rates  and  a  greater  concentration  of  poverty  than  suburban 
or  rural  communities;  thus,  insights  gathered  from  these  studies  will  be 
useful  in  understanding  the  potential  for  the  success  of  welfare  reform  in 
the  event  of  an  economic  downturn.  For  example,  one  study — the  Three 
City  Study — will  survey  primarily  low-income,  single-mother  families  from 
poor  and  moderate-income  areas  in  Boston,  Chicago,  and  San  Antonio, 
with  half  of  those  surveyed  being  TANF  recipients.12  The  survey  will  collect 
information  on  adult  and  family  well-being,  employment,  and  welfare 
receipt  three  times  within  4  years. 

Finally,  a  body  of  welfare  reform  research  examines  the  implementation  of 
TANF  at  the  state  and  local  levels.  Since  PRWORA  has  not  only  granted 
states  greater  responsibility  for  providing  cash  assistance  but  also  changed 
the  nature  of  cash  assistance,  it  is  important  to  learn  how  states  and 
localities  are  coping  with  these  changes.  Much  of  the  research  about 
program  implementation  focuses  on  challenges  faced  by  state,  and  in  some 
cases  local,  administrators  in  implementing  TANF.  Typically,  in  this 
research  qualitative  data  are  collected  by  visiting  state  or  local  TANF 
agencies;  reviewing  program  records;  and  interviewing  agency  officials, 
caseworkers,  and  clients.  For  example,  the  State  Capacity  Study  conducted 
by  the  State  University  of  New  York,  Rockefeller  Institute  of  Government, 
is  collecting  data  in  20  states  about  the  implementation  of  TANF  at  the  state 
level,  such  as  the  structure  of  government  services  and  information 
systems  used  to  track  clients. 


'holms  Hopkins  University,  in  collaboration  with  the  University  of  Texas  at  Austin, 
Pennsylvania  State  University,  Northwestern  University,  and  Harvard  University,  is 
conducting  the  Three  City  Study. 
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Framework  for  Assessing 
Data  Availability 


Because  we  expect  much  of  the  reauthorization  debate  to  focus  on  TANF’s 
four  legislative  goals,  the  framework  for  our  data  assessment  was  based  on 
those  goals.  To  assess  whether  data  exist  to  address  the  goals,  we  first 
created  a  list  of  “descriptive”  and  “effect”  research  questions  relevant  to 
each  goal.  Descriptive  questions  concern  a  low-income  individual’s  or 
family’s  status  or  behavior,  such  as  the  receipt  of  TANF  cash  assistance  or 
support  services  like  transportation,  housing,  child  care,  or  health  services; 
an  adult’s  employment  status  and  earnings;  and  a  family’s  reliance  on  non- 
TANF  government  benefits,  such  as  Food  Stamps,  Medicaid,  or  the  Earned 
Income  Tax  Credit.  Effect  questions  concern  the  extent  to  which  changes 
in  an  individual’s  or  family’s  status  or  behavior,  such  as  obtaining 
employment,  earning  income,  avoiding  out-of-wedlock  births,  or  forming  a 
two-parent  family,  are  the  result  of  the  TANF  program.  These  research 
questions  represent  the  broad  issues  that  the  Congress  will  consider  during 
TANF’s  reauthorization. 


To  summarize  our  findings,  we  identified  data  categories  associated  with 
TANF’s  goals,  some  of  which  are  more  narrowly  focused  than  the  research 
questions.  The  data  categories  represent  combinations  of  topics  we  found 
in  the  data,  such  as  employment  and  earnings  or  family  and  child  well¬ 
being,  that  were  associated  with  the  research  questions.  Figure  1  shows  the 
relationships  among  TANF’s  goals,  the  research  questions,  and  the  data 
categories,  several  of  which  are  associated  with  more  than  one  question. 
We  then  compared  the  data  categories  with  the  HHS  administrative  data, 
the  data  collected  by  national  surveys,  and  the  data  derived  from  existing 
and  planned  studies. 

Our  assessment  of  the  data’s  usefulness  for  determining  TANF’s  progress  is 
based  on  the  data’s  strengths  and  weaknesses,  the  design  of  the  survey  or 
study  for  which  the  data  were  gathered,  and  the  topics  to  which  the  data 
related.  The  criteria  we  used  in  assessing  the  strengths  and  weaknesses  of 
survey  data  included  survey  sample  size,  the  attrition  rate  of  respondents 
from  whom  data  were  collected  over  time,  and  survey  response  rate.  For 
administrative  data,  we  examined  the  geographic  scope  and  the 
comparability  of  the  data  among  states.  The  design  features  examined 
included  what  the  data  collection  method  was,  whether  the  data  were 
collected  at  one  point  in  time  or  at  different  points  in  time,  and  whether  the 
data  were  used  for  descriptive  analysis  of  TANF  or  AFDC  program 
recipients  and  their  families  or  analysis  of  the  program’s  effects.  Data  that 
can  be  used  for  descriptive  analysis  are  useful  for  research  that  addresses 
questions  in  the  descriptive  column  of  figure  1,  and  data  that  can  be  used 
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for  analyses  of  effect  are  useful  for  questions  in  the  effect  column  of  the 
figure. 
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Figure  1 :  TANF’s  Legislative  Goals,  Research  Questions,  and  Data  Categories  Associated  With  Those  Goals 


TANF  Goals  Descriptive  Research  Questions 


Provide  Assistance 
to  Needy  Families  So  That 
Children  May  Be  Cared  for 
in  Their  Own  Homes  or  in 
the  Homes  of  Relatives 


2.  How  Has  the  TANF  Cash  Assistance 
Benefit  States  Pay  to  Families  Changed 
Since  Enactment  of  PRWORA? 


1 .  How  Have  the  Size  and  Composition  of 
the  TANF  Cash  Assistance  Caseload 
Changed  Since  Enactment  of  PRWORA? 


Data  Categories 


Change  in  Size  and 
Composition  of  Caseload 


Effect  Research  Questions 


3.  What  Types  of  Noncash  Assistance  Are 
State  TANF  Programs  Providing  to  Low- 
Income  Families? 


4.  To  What  Extent  Do  Former  Recipients 
Depend  on  Government  Benefits  Other 
Than  TANF? 


Current  or  Former  TANF 
Recipients’  Use  of 
Noncash  Assistance 


End  Dependence  of 
Needy  Parents  on 
Government  Benefits 
by  Promoting  Job 
Preparation,  Work, 
and  Marriage 


5.  To  What  Extent  Are  TANF  Work 
Requirements  and  Time  Limits  Being 
Met? 


•  Employment  and  Earnings 


6.  What  Strategies  Have  Been  Used  to 
Promote  Marriage  Among  Low-Income 
Families? 


•  Strategies  to  Promote 
Marriage 


•  Strategies  to  Promote  Work 

•  Employment  and  Earnings 


7.  What  Strategies  Have  Been  Effective  in 
Moving  Needy  Parents  Into  Work? 


•  Employment  and  Earnings 

•  Family  and  Child  Well-Being 


8.  What  Has  Been  the  Effect  of  TANF  Work 
Requirements  on  Family  Well-Being? 


•  Marital  Status 

•  Employment  and  Earnings 

•  Family  and  Child  Well-Being 


9.  To  What  Extent  Has  Marriage  Improved 
the  Well-Being  of  Low-Income  Families? 


Prevent  and  Reduce  the 
Incidence  of  Out-of- 
Wedlock  Pregnancies 


•  Strategies  to  Prevent 
Out-of-Wedlock  Pregnancies 

•  Out-of-Wedlock  Birthrates 


10.  What  Strategies  Have  Been  Effective  in 
Preventing  Out-of-Wedlock 
Pregnancies? 


Encourage  the  Formation 
and  Maintenance  of  Two- 
Parent  Families 


•  Strategies  to  Promote  Family 
Formation 

•  Marital  Status 


1 1 .  What  Strategies  Have  Been  Effective  in 
Promoting  the  Formation  and 
Maintenance  of  Two-Parent  Low-Income 
Families? 
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National,  State,  and 
Local  Data  Address 
Different  TANF  Goals 


Together,  national  surveys,  HHS  administrative  data,  and  data  from  state 
and  local  studies  of  welfare  reform  address  TANF’s  four  legislative  goals.13 
The  national  data  provide  extensive  information  related  to  TANF’s  goals  of 
providing  assistance  to  needy  families  and  ending  dependency  on 
government  benefits  through  job  preparation,  work,  and  marriage.  State 
and  local  data  not  only  address  the  same  goals  as  the  national  data  but  in 
some  cases  also  provide  information  related  to  the  goals  of  preventing  out- 
of-wedlock  pregnancies  and  promoting  family  formation. 


National  Data  Address  Two 
TANF  Goals  but  Provide 
Limited  Information  on  Out- 
of- Wedlock  Pregnancy 
Prevention  and  Family 
Formation  Strategies 


National  data  provide  detailed  descriptive  information  related  to  two  of 
TANF’s  goals,  but  limited  information  related  to  TANF’s  goals  of  preventing 
out-of-wedlock  pregnancies  and  promoting  family  formation.  HHS 
administrative  data14  and  the  six  national  surveys  we  examined — the  CPS, 
NLSY,  NSAF,  PSID,  SIPP,  and  SPD — provide  descriptive  information  related 
to  TANF’s  goal  of  providing  assistance  to  needy  families,  including 
information  about  the  change  in  size  and  composition  of  the  TANF 
caseload  and  the  use  of  noncash  assistance  by  current  and  former  TANF 
recipients  (see  fig.  2). 15 


13See  app.  II  for  a  complete  list  of  the  national  surveys  and  studies  about  welfare  reform  that 
we  examined  for  data. 

14HHS  administrative  data  are  data  that  states  report  to  HHS  on  their  AFDC  and  TANF 
caseloads. 

16We  reviewed  only  the  CPS  March  Supplement. 
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Figure  2:  National  Data  That  Address  TANF’s  Goals  and  Related  Data  Categories 


Goal  1 

Goal  2 

Goal  3 

Goal  4 

Type  of 
Data 

Provide  Assistance  to  Needy 
Families  So  That  Children 

May  Be  Cared  for  in  Their 
Own  Homes  or  in  the  Homes 
of  Relatives 

End  Dependence  of  Needy  Parents 
on  Government  Benefits  by 
Promoting  Job  Preparation,  Work, 
and  Marriage 

Prevent  and  Reduce  the 
Incidence  of  Out-of- 
Wedlock  Pregnancies 

Encourage  the 
Formation  and 
Maintenance  of 
Two-Parent  Families 

Change  in 
Size  and 
Composition 
of  Caseload 

Current  or 
Former 
TANF 

Recipients’  Use 
of  Noncash 
Assistance 

Strategies 
to  Promote 
Work 

Employment 
and  Earnings 

Family 

and 

Child 

Well- 

Being 

Strategies 
to  Prevent 
Out-of- 
Wedlock 
Pregnancies 

Out-of- 

Wedlock 

Birthrates3 

Strategies 
to  Promote 
Family 
Formation 

Marital 

Status 

HHS 

Administrative 

Datab 

• 

• 

• 

• 

• 

•  C 

• 

•  C 

• 

Current 
Population 
Survey  (CPS) 

• 

• 

• 

• 

• 

National 
Longitudinal 
Survey  of 

Youth  (NLSY) 

• 

• 

• 

• 

• 

• 

National 

Survey  of 
American 
Families 
(NSAF) 

• 

• 

• 

• 

• 

• 

Panel  Study 
of  Income 
Dynamics 
(PSID) 

• 

• 

• 

• 

• 

• 

Survey  of 
Income  and 
Program 
Participation 
(SIPP) 

• 

• 

• 

• 

• 

• 

Survey  of 
Program 
Dynamics 
(SPD) 

• 

• 

• 

• 

• 

• 

Note:  We  did  not  assess  the  American  Community  Survey,  a  national  survey  that  will  collect 
information  about  welfare  receipt  and  other  measures,  because  the  data  will  not  be  available  in  time  for 
the  reauthorization  debate.  This  survey  was  developed  by  the  Census  Bureau  as  a  way  to  provide 
more  timely  demographic  information  for  local,  state,  and  federal  governments.  The  Census  Bureau  is 
currently  testing  the  data  collection  instrument  and  sampling  strategy  for  this  survey,  and  full-scale 
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implementation  for  a  sample  of  3  million  households  is  scheduled  for  fiscal  year  2003.  We  also  did  not 
assess  the  National  Survey  of  Family  Growth,  which  collects  information  on  fertility,  family  formation, 
and  other  family  characteristics,  because  its  post-TANF  data  will  not  be  available  in  time  for 
reauthorization.  The  National  Survey  of  Family  Growth  is  administered  by  the  National  Center  for 
Health  Statistics. 

aOut-of-wedlock  birthrates  may  be  able  to  be  constructed  from  such  measures  as  date  of  marriage  and 
child's  birthdate,  which  many  of  the  national  surveys  collect. 

bHHS  administrative  data  are  data  reported  by  the  states  about  AFDC  or  TANF  caseloads. 

“Beginning  in  fiscal  year  2000,  each  state  must  include  this  information  in  its  annual  report.  The  first 
fiscal  year  2000  report  was  due  December  31 ,  2000. 

National  data  also  address  TANF’s  goal  of  ending  dependence  on 
government  benefits  by  describing  the  circumstances  of  those  receiving 
TANF  and  those  who  are  no  longer  receiving  TANF.  HHS  administrative 
records  and  national  surveys  provide  descriptive  information  about  TANF 
recipients’  participation  in  work  activities,  employment  status,  earnings, 
and  other  family  well-being  measures.  HHS  administrative  records  contain 
information  only  about  whether  a  recipient  is  working  and  how  much 
income  that  individual  earns,  while  national  surveys  collect  more  detailed 
employment  and  earnings  data,  such  as  the  types  of  jobs  held  and  the 
hourly  wage.  National  data  are  also  available  about  family  well-being 
measures,  which  provide  information  about  how  TANF’s  focus  on  work 
and  marriage  may  be  changing  the  lives  of  low-income  families.  For 
instance,  national  surveys  have  information  about  the  amount  of  personal 
income  spent  on  health  and  housing,  whether  recipients  or  former 
recipients  rent  or  own  housing,  and  the  well-being  of  children  of  welfare 
recipients.  Several  of  the  national  surveys  provide  information  about 
children’s  school  attendance  or  developmental  status,  while  SIPP  and  SPD 
also  collect  data  about  the  number  of  births  to  teenagers.  SIPP  is  the  only 
national  survey  we  examined  that  contains  information  about  whether 
parents  have  had  to  terminate  their  parental  rights  or  give  a  child  up  for 
adoption. 

National  data  related  to  the  goals  of  preventing  out-of-wedlock  pregnancy 
and  promoting  family  formation  are  limited.  While  all  the  national  data  sets 
include  information  about  recipients’  and  nonrecipients’  marital  status, 
only  HHS  administrative  records  contain  information  about  out-of-wedlock 
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births  among  the  TANF  caseload.16  However,  states  did  not  begin  reporting 
this  information  to  HHS  until  fiscal  year  2000. 

Aside  from  information  about  welfare  reform  in  general,  national  surveys 
and  HHS  collect  information  about  several  different  groups  of  individuals 
affected  by  TANF,  including  those  who  remain  on  assistance,  those  who  no 
longer  receive  TANF,  those  who  are  diverted  from  TANF,17  and  those  who 
are  eligible  but  choose  not  to  participate.  HHS  administrative  data  and  all 
six  national  surveys  collect  data  about  current  and  former  TANF  recipients, 
but  the  type  of  information  collected  about  these  individuals  differs.  As 
figure  3  shows,  only  the  NSAF  and  SIPP  have  data  about  those  diverted 
from  TANF,  while  the  NLSY,  NSAF,  PSID,  SIPP,  and  SPD  have  data  about 
individuals  who  are  eligible  to  receive  TANF  but  do  not. 


16HHS  has  tracked  changes  in  the  number  of  out-of-wedlock  pregnancies  in  each  state  with 
data  collected  by  the  National  Center  for  Health  Statistics,  Division  of  Vital  Statistics.  HHS 
uses  this  information  to  award  bonuses  to  states  that  have  decreased  their  number  of  out-of- 
wedlock  pregnancies. 

17Under  TANF,  states  may  opt  to  offer  cash  diversion  payments  or  job  search  services  to 
families  applying  for  TANF  benefits.  Diversion  payments  and  services  are  intended  to 
address  an  emergency  situation  and  keep  families  from  entering  the  welfare  system. 
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Figure  3:  National  Survey  and  HHS  Administrative  Data  on  TANF  Groups 
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aThe  actual  year  varies  for  each  survey  and  HHS  administrative  data  set  reviewed. 
bSIPP  collects  information  on  total  benefits  received  in  a  month. 

°As  part  of  TANF  reporting  requirements,  states  must  report  this  information  to  HHS  beginning  in  fiscal 
year  2000.  States  also  report  information  on  non-TANF  assistance  to  TANF  recipients,  such  as 
Medicaid,  housing,  and  Supplemental  Security  Income. 

dSpecial  Supplemental  Nutrition  Program  for  Women,  Infants,  and  Children. 


State  and  Local  Data 
Complement  and  Fill  Gaps 
in  the  National  Data 


The  state  and  local  data  we  reviewed  can  be  classified  into  four  categories 
that  complement  and,  in  some  cases,  fill  in  gaps  not  covered  by  the  national 
data.  Waiver  data  come  from  evaluations  that  tested  the  effects  of 
programs  implemented  by  states  under  waivers  approved  by  HHS  prior  to 
TANF.18  Demonstration  data  come  from  studies  that  tested  the 
effectiveness  of  particular  strategies  aimed  at  individuals  either  receiving 
welfare  or  eligible  to  receive  welfare.  Leavers  data  come  from 
administrative  records  and  surveys  that  describe  the  circumstances  of 
those  who  left  welfare.  Finally,  metropolitan  and  community-based  data 
come  from  studies  that,  in  general,  describe  the  circumstances  of  low- 
income  families  and  TANF  participants  in  specific  metropolitan  areas, 
neighborhoods,  or  communities. 


Waiver  data  have  been  used  to  examine  the  effects  of  TANF-like  provisions 
on  welfare  recipients’  employment  status,  birth  rates,  and  marital  status,  as 
shown  in  figure  4.  Several  states  have  been  evaluating  the  waiver 
provisions  in  their  welfare  programs  by  randomly  assigning  welfare 
recipients  to  either  the  waiver  program  or  AFDC.  Waiver  programs  require 
participants  to  follow  provisions  that  later  were  required  or  permitted 
under  TANF,  such  as  being  required  to  work  or  risk  losing  eligibility  for 
benefits  or  being  allowed  to  receive  welfare  for  only  a  limited  time.  Most  of 
the  waiver  program  evaluations  collected  data  used  to  analyze  the  effect  of 
waivers  on  welfare  receipt,  employment,  and  income.  Data  from  several  of 
the  evaluations  have  also  been  used  to  analyze  the  effects  of  waivers  on 
out-of-wedlock  pregnancy  or  family  formation. 


lsUnder  section  1115  of  the  Social  Security  Act,  HHS  was  authorized  to  grant  states  waivers 
of  statutory  requirements  governing  the  AFDC  program.  This  authority  was  intended  to  give 
states  the  flexibility  to  test  innovations  in  their  programs,  such  as  limiting  the  length  of 
benefit  receipt  or  strengthening  work  requirements. 
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Figure  4:  State  and  Local  Data  That  Address  TANF’s  Goals  and  Related  Data  Categories 
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Note:  A  dot  indicates  that  at  least  one  data  set  provides  information  related  to  this  issue. 

“Data  related  to  these  topics  were  collected  from  demonstration  programs  operated  before  enactment 
of  PRWORA  in  1996. 

With  the  passage  of  PRWORA,  several  states  incorporated  their  waiver 
provisions  into  their  TANF  program  and  have  been  collecting  data  about 
the  experiences  of  participants  in  the  program.  Some  of  these  states  chose 
not  to  continue  their  evaluations  as  originally  designed,  instead  conducting 
modified  evaluations  that  typically  involved  studies  that  will  provide 
information  on  the  experience  of  implementing  the  program.  For  example, 
Montana  is  surveying  TANF  participants  to  collect  data  about  the  duration 
of  their  welfare  receipt,  the  types  of  noncash  assistance  they  use,  and  their 
employment. 


Demonstration  data  provide  information  on  topics  that  are  similar  to  those 
addressed  by  waiver  data  and  have  also  been  used  to  analyze  the  effects  of 
programs  on  their  participants,  but  demonstration  data  differ  in  two  key 
ways.  First,  most  demonstration  data,  including  all  data  related  to 
pregnancy  prevention  and  family  formation,  were  collected  before 
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PRWORA  was  enacted.  Second,  demonstration  data  were  collected  for 
studies  focused  on  how  a  particular  approach  affected  program 
participants.  In  fact,  many  of  the  demonstration  data  we  examined  were 
used  entirely  to  assess  the  effects  of  various  strategies  on  participants’ 
employment  status  and  earnings,  which  helps  to  distinguish  the  effects  of 
particular  provisions  included  in  a  program  like  TANK 

Leavers  data  provide  descriptive  information  about  those  who  have  left 
welfare.  This  information  includes  the  length  of  time  an  individual  received 
TANF,  reasons  for  leaving  welfare,  types  of  noncash  assistance  used,  and 
employment  and  earnings  information.  In  addition,  some  leavers  data  sets 
contain  information  about  former  recipients’  marital  status,  and  a  few  have 
data  about  the  number  of  pregnancies  and  births  among  former  recipients. 

Metropolitan  and  community-based  data  cover  some  of  the  same  issues  as 
the  other  data  categories,  including  information  about  TANF  work 
requirements  and  time  limits.  Although  the  same  issues  are  addressed,  the 
data  are  collected  in  large  cities  or  neighborhoods  in  order  to  examine  the 
circumstances  of  welfare  recipients  in  areas  that  may  have  high 
concentrations  of  poverty  or  limited  access  to  jobs.  In  addition, 
metropolitan  and  community-based  data  provide  information  about  groups 
other  than  TANF  recipients  and  former  recipients — including  individuals 
diverted  from  TANF  and  those  who  are  eligible  to  participate  in  TANF  but 
do  not. 


Strengths  and 
Weaknesses  of  Existing 
Data  Affect  Usefulness 
for  Assessing  TANF’s 
Progress 


Although  existing  data  provide  rich  information  about  the  lives  of  families 
who  are  receiving  or  have  received  TANF,  the  strengths  and  weaknesses  of 
these  data  affect  their  usefulness  for  understanding  welfare  under  the 
TANF  block  grant.  National  data  can  be  analyzed  to  gain  a  descriptive 
picture  of  what  has  happened  under  TANF  for  the  nation  as  a  whole. 
However,  of  the  seven  national  data  sets  we  reviewed,  only  two  can  be  used 
to  describe  the  well-being  of  families  receiving  TANF  within  individual 
states.  Although  waiver  and  demonstration  data  can  be  analyzed  to  gain 
information  about  TANF’s  effects,  these  analyses  can  be  done  within  only  a 
limited  number  of  states  and  disparate  localities.  We  examined  nearly  40 
data  sets  that  could  be  analyzed  for  information  about  the  circumstances  of 
former  recipients.  However,  only  a  subgroup  of  these  data  sets  met  criteria 
that  allowed  the  sample  to  be  generalized  statewide.  These  data  sets 
represented  15  states.  In  some  cases,  the  value  of  survey  data  collected 
from  those  who  left  welfare  was  limited  because  few  former  recipients 
actually  responded  to  the  surveys:  in  some  cases,  former  recipients  could 
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not  be  located,  and  in  other  cases  they  chose  not  to  answer  the  questions 
posed  to  them.  Metropolitan  and  community-based  data  can  be  analyzed  to 
describe  changes  over  time  in  the  lives  of  welfare  recipients  in  urban 
centers.  Much  of  this  data  collection  will  continue  beyond  2001. 


National  Survey  Data 
Provide  Descriptive 
Information  About  Nation, 
but  Not  Effects  in  Individual 
States 


The  strength  of  the  national  data  is  that  they  were  collected  from  samples 
selected  randomly  from  the  nation’s  population  and  include  low-income 
families  and  TANF  recipients  in  numbers  sufficient  to  allow  reliable 
estimates  about  these  groups.  In  addition,  most  of  the  national  data  were 
collected  for  the  same  individuals  over  time,  allowing  changes  in  welfare 
recipients’  employment,  earnings,  and  well-being  to  be  tracked  across 
programs  implemented  at  different  times.  However,  all  the  national  surveys 
have  participants  who  drop  out  of  the  survey  sample  over  time,  and  this 
may  limit  how  well  the  samples  represent  the  nation’s  welfare  recipients. 


National  data  are  collected  from  random  samples  that  contain  low-income 
families  and  TANF  recipients.  Because  samples  from  national  surveys  are 
selected  randomly,  they  are,  at  the  time  of  selection,  representative  of  the 
population  at  large,  including  the  welfare  population.  In  addition,  all  the 
national  data  sets  we  reviewed  have  sample  sizes  large  enough  to  allow 
reliable  estimates  about  the  nation’s  low-income  and  TANF  populations — 
as  sample  size  increases,  the  degree  of  precision  of  the  estimates  made 
using  that  sample  also  increases  (see  table  1). 


Table  1 :  Overall  Sample  Sizes  for  National  Data  Sources 

National  data  sources 

Overall  sample  size 

HHS  administrative  data3 

164,481 

CPS 

50,000 

NLSY 

12,686 

NSAF 

46,000 

PSID 

7,000 

SIPP 

36,800 

SPD 

37,000 

aUnder  the  TANF  data  reporting  system,  states  have  the  option  to  submit  either  sample  data  or  data  on 
their  entire  TANF  caseload  to  FIHS.  For  its  fiscal  year  1999  report  to  the  Congress,  HHS  randomly 
selected  approximately  200  to  400  cases  each  month  from  each  state  that  submits  data  on  its  entire 
caseload.  Thus,  the  total  in  column  two  combines  data  from  states  that  submitted  a  sample  and  states 
that  submitted  data  on  the  entire  caseload. 
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As  shown  in  figure  5,  two  national  data  sources  collect  data  on  individuals 
at  one  point  in  time;  others  collect  data  on  the  same  individuals  across 
time.  In  both  cases,  the  data  can  be  used  for  comparisons  between  groups 
of  individuals  living  under  welfare  provisions  implemented  at  different  time 
periods. 


Figure  5:  Data  Sources  That  Collect  Information  at  One  Point  in  Time  and  Across 
Time 
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aThe  CPS  rotates  participants  in  the  following  way:  a  housing  unit  is  interviewed  at  regular  intervals  for 
4  consecutive  months;  for  the  next  8  months,  the  housing  unit  is  not  part  of  the  sample;  the  unit  is  then 
interviewed  for  the  next  4  months  and  then  retired  from  the  sample.  As  a  result,  less  than  50  percent  of 
the  sample  can  be  linked  across  years. 

Five  national  surveys — the  CPS,  NLSY,  PSID,  SIPP,  and  SPD — collect  data 
from  the  same  individuals  over  time.  For  the  SIPP,  the  Census  Bureau,  after 
a  specified  period,  changes  the  group  of  individuals  from  whom  data  are 
collected.  For  example,  the  1993  SIPP  panel  followed  a  group  of  individuals 
through  1996.  In  1996,  a  new  group  was  randomly  selected  and  followed 
through  2000.  Data  collected  over  time  could  be  analyzed  to  describe  how 
people  cycle  on  and  off  TANF,  how  their  use  of  benefits  changes  over  time, 
and  how  their  family  well-being  changes.  In  addition,  comparisons  could  be 
made  between  groups  covered  by  different  welfare  provisions.  For 
example,  AFDC  recipients  included  in  the  1993-96  SIPP  panel  could  be 
compared  with  TANF  recipients  who  were  part  of  the  1996-2000  SIPP 
panel.  The  NSAF,  as  well  as  HHS  administrative  records,  has  collected  data 
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from  different  samples  of  individuals  in  different  years.  For  example,  in 
1997  one  group  of  people  completed  the  NSAF;  another  group  completed 
the  survey  in  1999.  In  cases  such  as  these,  the  samples  from  different  years 
can  be  compared  with  each  other  to  look  for  changes  across  time. 

For  those  national  surveys  that  collect  information  about  changes  in 
welfare  across  time,  the  likelihood  that  survey  participants  will  drop  out 
over  time  increases,  potentially  affecting  how  well  the  data  actually 
represent  all  members  of  the  nation’s  low-income  and  TANF  populations. 
In  general,  the  greater  the  attrition  rate,  the  less  likely  a  sample  is  to  be 
representative  of  the  larger  population  from  which  it  was  drawn.  Those 
who  have  continued  participating  in  the  survey  may  be  different  from  those 
who  stopped  or  dropped  out.  As  surveys  that  collect  data  over  time,  the 
NLSY,  PSID,  SIPP,  and  SPD  all  have  experienced  sample  loss,  as  shown  in 
table  2.  Concerns  about  attrition  are  especially  significant  for  the  SPD, 
because  it  was  designed  specifically  to  track  welfare  recipients  from  AFDC 
through  TANF.  Census  has  tried  mathematically  adjusting  available 
responses  to  compensate  for  the  survey’s  sample  loss,  but  this  adjustment 
has  not  sufficiently  remedied  the  problem,  according  to  a  Census  official. 
Census  will  take  steps  to  lessen  attrition  through  intensive  follow-up  with 
survey  dropouts  to  enlist  their  participation  and  through  the  use  of 
monetary  incentives  for  future  respondents  to  the  survey. 


Table  2:  Attrition  Rates  for  National  Data  Sources  That  Collect  Data  Over  Time 


Survey 

Attrition8  calculated  from... 

Percentage  of 
sample  that 
stopped 
participating 

Beginning  of  data 
collection 

Through 

NLSY79 

1979 

1996 

36 

PSID 

1968 

1993 

47 

SPD 

1992 

1998 

50 

SIPP 

1996 

2000 

36 

Note:  Attrition  rates  vary  depending  on  whether  deceased  respondents  are  included  in  the  calculation. 


aThe  attrition  rates  for  the  four  national  surveys  that  collect  data  across  time  are  not  necessarily 
comparable.  As  the  table  shows,  some  surveys  started  tracking  their  samples  earlier  than  others, 
increasing  the  potential  for  sample  loss.  In  addition,  surveys  vary  in  when  they  begin  counting  attrition. 
Some  start  counting  attrition  only  after  the  sample  has  been  formed,  while  others  include  in  the 
calculation  those  who  were  selected  for  the  original  sample  but  who  could  not  be  contacted  or  did  not 
participate.  See  Daniel  H.  Weinberg  and  Stephanie  S.  Shipp,  The  Survey  of  Program  Dynamics:  A 
Mid-term  Status  Report  (Washington,  D.C.:  U.S.  Census  Bureau,  Feb.  2,  2000)  for  a  more  detailed 
discussion  of  attrition  and  a  related  measure,  cumulative  response  rate,  in  the  SPD,  NLSY,  and  PSID. 
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For  national  surveys,  the  response  rate — the  number  of  people  in  the 
survey  sample  who  actually  responded,  compared  with  those  who  were 
asked  to  respond  but  did  not — has  been  large  enough  to  allow  the  survey 
results  to  be  generalized  beyond  those  who  completed  the  survey,  with  the 
exception  of  the  1999  NSAF.  Most  practitioners  of  survey  research, 
including  GAO,  require  at  least  a  70-  to  75-percent  response  rate  before 
survey  data  can  be  generalized  beyond  those  who  completed  the  survey.  As 
table  3  shows,  the  response  rate  for  all  the  national  surveys  except  the  1999 
NSAF  was  at  or  above  the  70-percent  standard.19  Given  the  survey’s 
response  rate,  using  the  1999  NSAF  survey  data  would  require  determining 
whether  patterns  in  who  responded  and  who  did  not  respond  existed  and 
what  this  means  for  how  well  respondents  represent  the  original  sample. 
For  those  surveys  that  collect  data  on  the  same  individuals  over  time, 
response  rates  sometimes  are  considered  in  conjunction  with  rates  of 
attrition. 


Table  3:  Response  Rates  for  National  Surveys 

National  surveys 

Response  rate 
(percentage) 

CPS 

84 

NLSY 

90a 

NSAF  1997 

70 

NSAF  1999 

64 

PSID 

92a 

SIPP 

90b 

SPD 

89a 

aThis  rate  represents  only  the  most  recent  administration  of  the  survey. 

bThe  typical  response  rate  for  the  1996  SIPP  panel  was  in  the  high  80s  to  low  90s. 

The  major  limitation  of  most  existing  national  data  is  that  they  cannot  be 
used  for  state-level  analyses.  In  general,  national  data  sources  have  state 
sample  sizes  that  are  too  small  to  allow  reliable  generalizations  about 


10The  NSAF’s  sample  was  gathered  by  a  procedure  known  as  random  digit  dialing  (RDD).  An 
RDD  sample  is  drawn  by  randomly  selecting  entries  from  a  list  of  telephone  numbers  for  the 
general  population.  Depending  on  how  recently  the  list  was  updated,  many  of  the  entries 
may  not  be  for  working  telephones.  Thus,  the  response  rates  of  RDD  samples  tend  to  be 
lower. 


Page  26 


GAO-Ol-298  Data  for  Assessing  TANF’s  Progress 


TANF  recipients  within  individual  states.  The  NLSY,  PSID,  SIPP,  and  SPD 
collect  data  not  from  states  per  se,  but  from  regions  that,  in  some  cases, 
include  more  than  one  state.  Thus,  while  these  data  can  be  analyzed  to 
provide  a  descriptive  picture  of  TANF  for  the  nation,  they  cannot  be  used 
within  states  for  descriptive  analyses  or  to  analyze  the  effects  of  states’ 
TANF  provisions.  This  does  not  mean  that  researchers  do  not  use  these 
data  sources  for  state-level  analyses.  For  example,  some  researchers 
combine  several  years  of  CPS  data  to  obtain  adequate  sample  sizes  within 
states  for  state-level  analyses.  However,  Census,  which  administers  the 
CPS,  SIPP,  and  SPD,  does  not  recommend  using  data  from  these  surveys 
for  state-level  analyses,  because  doing  so  when  sample  sizes  are  small  may 
produce  findings  that  are  not  reliable.20 

Two  national  data  sources,  HHS  administrative  records  and  the  NSAF 
survey,  can  be  used  for  state-level  analysis,  but  with  limitations.  HHS 
administrative  records  provide  data  from  all  50  states  and  the  District  of 
Columbia.  However,  the  reporting  requirements  for  these  data  are  not 
completely  standardized  across  states,  so  that  how  a  variable  is  defined 
may  vary  among  states.  For  example,  each  state  may  define  the  work  or 
work-related  activities  in  which  TANF  recipients  participate  as  they  think 
appropriate  to  the  state  program. 

Like  HHS  administrative  records,  NSAF  survey  data  can  be  used  for  state- 
level  analyses.  NSAF  has  samples  large  enough  to  allow  state-level  analyses 
in  13  states,21  representing  58  percent  of  the  fiscal  year  1999  national  TANF 
caseload;  this  is  not  the  case  in  the  37  remaining  states.  For  example,  the 
number  of  low-income  children  surveyed  for  the  1997  NSAF  ranged  from  a 
low  of  760  to  a  high  of  1,813  in  each  of  the  13  states  where  NSAF  collected 
samples  large  enough  to  permit  state-level  analysis.  However,  the  number 
of  low-income  children  surveyed  in  the  37  remaining  states  averaged  35  per 
state,  a  number  too  small  to  allow  reliable  conclusions  about  the  children 
of  TANF  recipients  in  any  of  these  states. 

Even  if  the  issue  of  sample  sizes  within  states  were  resolved,  obstacles  to 
using  the  national  data  to  analyze  TANF’s  effects  within  states  would  still 


20It  should  be  noted  that  Census  will  expand  the  sample  of  the  CPS  in  2001.  The  sample  size 
will  be  doubled,  which  will  increase  sample  sizes  in  individual  states  and  allow  improved 
state-level  estimates. 

21  Alabama,  California,  Colorado,  Florida,  Massachusetts,  Michigan,  Minnesota,  Mississippi, 
New  Jersey,  New  York,  Texas,  Washington,  and  Wisconsin. 
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exist.  The  lack  of  information  about  the  choices  states  have  made  about 
TANF  policies  and  program  rules  has  been  identified  as  one  of  the 
challenges  to  using  national  data  to  analyze  TANF’s  effects.  However, 
research  organizations  have  collected  this  information.  The  Center  for  Law 
and  Social  Policy  has  worked  with  the  Center  on  Budget  and  Policy 
Priorities  to  document  policies  in  all  50  states  and  the  District  of  Columbia, 
and  the  Urban  Institute  has  developed  a  state  database  that  documents 
state  program  rules.  Yet,  even  with  this  information,  using  national  data  to 
measure  state-level  effects  poses  challenges. 

The  first  challenge  is  deciding  with  whom  TANF  recipients  should  be 
compared.  To  test  TANF’s  effects,  the  employment,  earnings,  and  well¬ 
being  of  individuals  in  the  program  must  be  compared  with  those  of 
individuals  who  are  not  in  the  program.  In  the  case  of  TANF,  it  would  be 
difficult  to  determine  what  group  should  provide  the  point  of  comparison.22 
Because  waivers  introduced  TANF-like  policies  and  program  rules  while 
AFDC  was  still  in  effect,  it  would  be  difficult  to  select  a  group  of  welfare 
recipients  whose  experiences  with  welfare  were  not  influenced  by  TANF. 

The  second  challenge  is  determining  the  effect  of  any  single  welfare 
provision  given  the  multiple  provisions  that  make  up  states’  TANF 
programs.  For  example,  TANF  recipients  are  required  to  work,  and  states 
must  impose  penalties  or  sanctions  when  recipients  do  not  comply  with 
work  requirements.  In  such  cases,  it  would  be  difficult  to  separate  the 
combined  effects  of  work  requirements  and  any  penalties  or  sanctions  that 
were  imposed  into  the  individual  effects  of  each. 

A  third  challenge  is  detecting  the  long-term  effects  of  state  programs  that 
have  been  recently  implemented.  Although  PRWORA  was  enacted  in  1996, 
states  implemented  their  TANF  programs  at  different  points  in  time.  Some 
states  were  still  refining  their  TANF  programs  at  the  beginning  of  1998. 
Consequently,  the  long-term  effects  of  TANF  may  not  yet  be  realized. 

Finally,  state-level  analyses  may  not  be  the  best  way  to  measure  TANF’s 
effects  in  every  state.  Some  states  have  further  devolved  TANF  to  localities, 
and  different  localities  may  implement  a  state’s  TANF  provisions 
differently.  In  total,  17  states  have  given  local  governments  responsibility 
for  TANF  program  design  and  implementation. 


“In  program  evaluation  research,  choosing  a  group  with  which  participants  in  a  program 
can  be  compared  is  referred  to  as  “selecting  the  counterfactual.” 
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Waiver  and  Demonstration 
Data  Allow  Assessment  of 
TANF  Effects,  but  Only  for 
Localities 


The  strength  of  the  waiver  and  demonstration  data  is  that  they  can  be  used 
to  analyze  TANF’s  effects,  but  with  few  exceptions  these  data  were 
collected  from  city  and  county  samples  rather  than  statewide  samples.  (See 
app.  II  for  the  localities  examined.)  Most  of  the  waiver  and  demonstration 
data  were  collected  as  part  of  experiments — studies  that  randomly 
assigned  welfare  recipients  to  groups  that  were  subject  to  different  welfare 
provisions.  Experiments,  when  done  correctly,  are  recognized  as  the  most 
rigorous  way  of  determining  the  extent  to  which  an  observed  outcome  can 
be  attributed  to  the  program  itself,  rather  than  to  differences  among  the 
program  participants.  Over  half  of  the  waiver  data  sets  and  virtually  all  of 
the  demonstration  data  sets  we  reviewed  consisted  of  data  from 
experiments.  Of  the  waiver  data  sets,  about  half  were  collected  from  city 
and  county  samples,  with  the  others  being  collected  from  statewide 
samples.  All  of  the  demonstration  data  sets  were  collected  from  city  and 
county  samples.  Overall,  6  of  the  54  waiver  and  demonstration  data  sets 
that  could  be  used  for  analyses  of  effect  were  collected  from  statewide 
samples.23 


According  to  the  project  directors  of  two  waiver  evaluations,  the  high  cost 
of  conducting  rigorous  program  evaluations  may  explain,  in  part,  why  data 
sets  used  to  analyze  TANF’s  effects  tend  to  use  samples  from  cities  and 
counties  and  not  entire  states.  Given  limited  resources,  researchers  may 
choose  to  conduct  rigorous  evaluations  in  selected  cities  or  counties  rather 
than  sacrifice  rigor  to  evaluate  a  program  statewide.  Data  sources  we 
reviewed  for  both  the  Vermont  and  Iowa  waiver  evaluations  mentioned 
budget  constraints  as  a  factor  that  led  researchers  to  limit  their  data 
collection  efforts. 


Another  limitation  of  the  waiver  and  demonstration  data  is  that  most  often 
they  were  collected  prior  to  the  implementation  of  TANF.  This  is  not 
surprising  given  that  in  many  cases  the  waiver  provisions  and  the 
demonstration  projects  were  intended  to  test  provisions  before  they  were 
adopted  and  implemented.  However,  the  provisions  tested  may  not  have 
been  those  ultimately  adopted  by  the  state. 

Finally,  in  almost  all  cases  in  which  waiver  evaluations  and  demonstration 
projects  collected  survey  data,  response  rates  were  above  the  70-percent 
standard  (see  table  4). 


ASee  app.  Ill  for  a  list  of  waiver  and  demonstration  data  sets  showing  the  geographic:  scope 
of  the  sample  and  the  type  of  analysis  for  which  each  data  set  could  be  used. 
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Table  4:  Number  of  Waiver  and  Demonstration  Data  Sets  With  Acceptable  Response 

Rates 

Total  waiver  and  demonstration  data 
sets  collected  using  surveys 

24 

Data  sets  with  response  rate  at  or  above  70  percent 

14 

Data  sets  with  response  rate  below  70  percent 

3 

Note:  In  seven  cases,  the  data  source  we  reviewed  did  not  report  a  response  rate. 


State-Level  Analyses  of 
Leavers  Data  Are  Possible  in 
15  States;  in  Some  Cases, 
Picture  Is  Incomplete 


The  strength  of  the  leavers  data  is  that  in  most  cases,  they  were  collected 
from  statewide  samples.  However,  in  some  cases,  leavers  data  collected 
using  surveys  may  not  be  representative  of  a  state’s  leaver  population. 
Although  we  reviewed  nearly  40  leavers  data  sets,  on  the  basis  of  the  type 
of  data  available,  response  rates,  and  the  absence  of  significant  differences 
between  survey  respondents  and  nonrespondents,  we  concluded  that  state- 
level  analyses  could  be  done  for  15  states  using  the  data  sets  we  examined. 


To  be  representative  of  a  state’s  leavers  population,  survey  data  need  to 
meet  the  70-percent  standard  for  response  rates,  or,  through  a  comparison 
of  survey  respondents  with  nonrespondents,  show  that  the  two  groups  do 
not  differ  significantly.  When  a  state  has  both  administrative  data  and 
survey  data  available,  the  administrative  data  could  be  used  in  place  of 
survey  data  that  are  not  representative.  As  figure  6  shows,  Arkansas, 
Florida,  Georgia,  North  Carolina,  and  South  Carolina  have  either  survey 
data  that  meet  the  standard  for  response  rates  or  data  from  survey 
respondents  who  were  not  significantly  different  from  nonrespondents. 
Arizona,  Colorado,  the  District  of  Columbia,  Illinois,  Kansas,  Missouri, 
Virginia,  Washington,  and  Wisconsin  have  both  administrative  data  and 
survey  data.  The  response  rate  for  the  District  of  Columbia,  Illinois,  Kansas, 
Virginia,  and  Wisconsin  was  below  70  percent,  but  for  Virginia,  a 
comparison  of  respondents  with  nonrespondents  revealed  no  significant 
differences  between  the  two  groups.  Although  New  York  has  no  survey 
data,  its  administrative  data  provide  information  about  the  state’s  leavers. 
California,  Massachusetts,  and  Texas  are  the  three  states  for  which,  given 
the  available  data,  state-level  analyses  of  leavers  cannot  be  done.  We 
previously  reported  that  eight  leavers  studies  covering  seven  states  had 
collected  adequate  information  to  allow  the  study  findings  to  be 
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generalized  to  the  states’  welfare  populations.24  Thus  4  states — Indiana, 
Maryland,  Oklahoma,  and  Tennessee — can  be  added  to  the  list  of  15  states 
we  identify  in  figure  6  as  having  data  that  can  be  generalized  statewide.20  In 
appendix  II  we  list  all  the  sources  we  reviewed  that  provide  data  on  those 
who  have  left  welfare. 


^GAO/I  IE  I IS-99-48 ,  Apr.  28,  1999. 

2r,Wc  examined  more  recent  leavers  data  for  three  of  the  seven  states  in  our  review. 
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Figure  6:  Leavers  Data  From  Statewide  Samples 


Administrative 

Data 

Survey 

Data 

Is  the  Survey 
Response  Rate 
70  Percent  or 
Higher? 

Do  the  Survey  Data 
Provide  Information 
Not  Included  in  the 
Administrative  Data? 

Arizona3 

• 

• 

Yes 

Information  About  the 

Total  Flousehold,  Hourly 
Wages,  Flours  Worked, 
Monthly  Earnings,  Food 
Shortages,  and  Other 
Material  Flardships 

Arkansas  3 

b 

• 

Yes 

b 

California 

b 

• 

No 

b 

Colorado3 

• 

• 

Yes 

No 

District  of 
Columbia3 

• 

• 

No 

Information  About  Use 
of  Noncash  Assistance 

Florida3 

b 

• 

Noc 

b 

Georgia3 

b 

• 

Noc 

b 

Illinois3 

• 

• 

No 

Information  About 

Whether  Work 

Requirements  and  Time 
Limits  Are  Being  Met; 

Size  and  Composition 
of  Caseload 

Kansas3 

• 

• 

No 

Information  About  Use 
of  Noncash  Assistance 

Massachusetts 

b 

• 

No 

b 

Missouri3 

• 

• 

Yes 

Information  About  the 

Total  Household,  Hourly 
Wages,  Hours  Worked, 
Monthly  Earnings,  Food 
Shortages,  and  Other 
Material  Hardships 

New  York  3 

• 

d 

d 

d 

North  Carolina3 

b 

• 

Yes 

b 

South  Carolina  3 

b 

• 

Yes 

b 

Texas 

b 

• 

No 

b 

Virginia3 

• 

• 

Noc 

Information  About  Use 
of  Noncash  Assistance; 
Benefits  Other  Than  TANF 

Washington3 

• 

• 

Yes 

Information  About  the 

Total  Household,  Hourly 
Wages,  Hours  Worked, 
Monthly  Earnings,  Food 
Shortages,  and  Other 
Material  Hardships 

Wisconsin3 

• 

• 

No 

Information  About  Use  of 
Noncash  Assistance 
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aLeavers  data  can  be  used  for  state-level  analyses. 
bOnly  survey  data  were  available. 

°A  comparison  of  respondents  with  nonrespondents  showed  no  significant  differences  between  the  two 
groups. 

dOnly  administrative  data  were  available. 

Some  researchers  may  wish  to  compare  those  who  left  TANF  with  those 
who  left  AFDC  on  outcomes  such  as  employment,  earnings,  and  well-being. 
Contrasting  outcomes  for  these  two  groups  would  require  deciding  which 
AFDC  leavers  provide  the  best  point  of  comparison.  Many  factors  specific 
to  the  year  in  which  recipients  left  the  welfare  rolls  would  influence  their 
employment  prospects,  wages,  and  well-being.  For  example,  labor  markets 
and  economic  conditions  in  a  given  year  would  influence  former  recipients’ 
employment  opportunities.  Historical  influences  such  as  these  would 
complicate  the  issue  of  selecting  a  comparable  group  of  AFDC  leavers  and 
TANF  leavers. 


Metropolitan  and 
Community-Based  Data 
Cover  Urban  Centers;  Data 
Collection  Will  Continue 
Beyond  2001 


The  strength  of  the  metropolitan  and  community-based  data  is  that  they 
can  be  used  in  descriptive  analyses  that  provide  information  about  how  the 
lives  of  low-income  families  and  TANF  participants  have  changed  over 
time.  Because  data  collection  is  occurring  over  time,  in  some  cases  it  has 
yet  to  be  completed.  For  example,  the  Los  Angeles  Family  and 
Neighborhood  Study  (LA  FANS)  is  collecting  data  about  participation  in 
welfare  programs  from  residents  of  65  neighborhoods  in  Los  Angeles 
County  over  a  4-year  period.  LA  FANS  began  data  collection  in  January 
2000  and  will  continue  data  collection  through  2004.  Most  of  the  materials 
we  reviewed  regarding  metropolitan  and  community-based  data  sets  did 
not  report  information  about  attrition  rates.  When  response  rates  were 
reported,  they  were  above  the  70-percent  standard.  Figure  7  shows  the  time 
periods  for  which  the  data  are  or  will  be  available  for  different 
metropolitan  areas  and  communities. 


Page  33 


GAO-Ol-298  Data  for  Assessing  TANF’s  Progress 


Figure  7:  Years  Covered  by  Data  Available  for  Metropolitan  and  Community  Areas 


Data  Source 

Years  Covered 
by  Data 
Collection 

Metropolitan  and  Community  Areas 

From  Which  Data  Are  Collected 

NICHD  Study  of  Early 

Child  Care 

1991-2004 

•  Little  Rock,  Arkansas  •  In  or  Near  Philadelphia  and 

•  Irvine,  California  Pittsburgh,  Pennsylvania 

•  Lawrence,  Kansas  •  Charlottesville,  Virginia 

•  Boston,  Massachusetts  •  Seattle,  Washington 

•  Morganton,  North  Carolina  •  Madison,  Wisconsin 

Fragile  Families  and 

Child  Well-Being  Study3 

1998-2004 

•  Oakland,  California  •  Philadelphia,  Pennsylvania 

•  Baltimore,  Maryland  •  Austin,  Texas 

•  Detroit,  Michigan  •  Richmond,  Virginia 

•  Newark,  New  Jersey 

LA  FANS 

2000-04 

•  Los  Angeles  County  Neighborhoods,  California 

Welfare,  Children  and  Families — 

A  Three  City  Study 

1997-2003 

•  Chicago,  Illinois  •  San  Antonio,  Texas 

•  Boston,  Massachusetts 

Project  on  Devolution  and 

Urban  Change:  An  Overview 
Assessing  Impact  of  Welfare 

Reform  on  Urban  Communities6 

•  Administrative  Records 

•  Community-Based  Indicators 

•  Survey  Data 

1992-2001 

1992-2001 

1998  and  2001 

•  Los  Angeles  County,  California  •  Cuyahoga  County,  Ohio 

•  Miami-Dade  County,  Florida  •  Philadelphia  County,  Pennsylvania 

Growing  Up  in  Poverty: 

Remember  the  Children: 

Mothers  Balance  Work  and 

Child  Care  Under  Welfare  Reform 

1998-2001 

•  San  Francisco,  California  •  New  Haven,  Connecticut 

•  Santa  Clara,  California  •  Tampa,  Florida 

Impact  of  Welfare  Reform 
on  Families 

1999-2000 

•  Milwaukee,  Wisconsin 

aln  total,  data  will  be  collected  from  20  cities. 

blnformation  for  the  three  data  sets  was  collected  from  all  four  sites. 

Three  of  the  metropolitan  and  community-based  data  sources  have 
measures  that  can  be  used  to  analyze  TANF’s  effects,  even  though  the  data 
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were  not  collected  as  part  of  an  experiment.26  For  example,  data  from  the 
Fragile  Families  study  can  be  used  to  examine  TANF’s  effects  by  drawing 
comparisons  between  the  3,675  unmarried  parents  and  the  1,125  married 
parents  who  compose  the  survey  sample  in  cities  with  populations  over 
200,000.  Data  collection  for  Fragile  Families  began  in  1998  and  will 
continue  through  2004.  The  data  have  already  been  used  to  examine 
differences  in  relationship  quality  between  married  and  unmarried  couples, 
including  whether  a  father  gave  money  to  or  helped  a  mother  in  a 
nonmonetary  way  during  pregnancy. 


Existing  Data  Are  More 
Adequate  for  State  and 
Local  Descriptive  Analyses 
Than  for  Analyses  of  Effect 


The  current  body  of  research  on  TANF  addresses  many  issues  of  interest  to 
the  Congress  but  does  not  provide  a  comprehensive  national  picture  of 
TANF.  However,  existing  national  data  and  data  from  state  and  local  studies 
could  be  pieced  together  to  develop  a  descriptive  picture  of  what  has 
happened  to  TANF  participants  in  all  50  states.  In  addition,  within  a  limited 
number  of  states  and  various  cities  and  counties,  existing  data  can  be  used 
to  conduct  analyses  of  TANF’s  effects. 


National  survey  data  can  be  used  with  data  from  HHS  administrative 
records  for  descriptive  analyses  of  TANF’s  progress  nationwide.  HHS 
administrative  data  can  be  used  for  analyses  within  each  of  the  50  states, 
and  national  survey  data  can  be  analyzed  for  national  trends.  These 
analyses  could  be  compared  to  examine  the  extent  to  which  the 
employment  experiences,  for  example,  of  current  and  former  TANF 
recipients  in  individual  states  conform  with  or  depart  from  the  experiences 
of  such  individuals  identified  with  national  survey  data.  This  comparison 
could  be  extended  to  the  individual  states  and  localities  covered  by  the 
NSAF  data,  waiver  and  demonstration  data,  leavers  data,  and  metropolitan 
and  community-based  evaluation  data. 


While'  piecing  the  data  together  in  this  way  would  build  on  their  strengths, 
each  data  type  still  has  limitations.  Specifically,  national  survey  data 
provide  national  samples  useful  for  comparing  the  lives  of  welfare 
recipients  covered  by  welfare  provisions  implemented  at  different  times. 


26These  three  studies — the  NICHD  Study  of  Early  Child  Care,  the  Fragile  Families  and  Child 
Well-Being  Study,  and  the  Project  on  Devolution  and  Urban  Change — were  designed  to 
approximate  experiments.  Rather  than  being  randomly  assigned  to  a  group,  individuals 
were  classified  into  groups  because  of  some  attribute  they  did  or  did  not  have.  When  studies 
are  designed  this  way,  having  or  not  having  the  attribute  is  believed  to  be  the  cause  of  any 
difference  in  outcomes  for  each  group. 
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However,  attrition  or  low  response  rates  may  affect  the  degree  to  which 
these  samples  represent  all  members  of  the  nation’s  low-income  and  TANF 
population.  Within  each  of  the  50  states,  HHS  administrative  data  can  be 
analyzed  to  gain  insight  into  current  recipients’  use  of  noncash  benefits, 
among  other  things,  but  the  lack  of  standardized  reporting  requirements 
would  complicate  comparisons  across  states.  Supplemental  descriptive 
analyses  for  individual  states  can  be  done  using  NSAF  survey  data,  leavers 
data,  waiver  and  demonstration  data,  and  metropolitan  and  community- 
based  data.  In  addition,  like  the  national  survey  data,  many  of  these  data 
represent  multiple  measures  over  time.  However,  these  analyses  in  many 
cases  can  be  generalized  only  to  cities  and  counties  and  not  to  entire  states. 

Existing  data  can  also  be  analyzed  to  gain  information  about  TANF’s 
effects.  Although  the  1997  and  1999  NSAF  survey  samples  do  not  include 
pre-TANF  welfare  recipients,  the  samples  do  include  other  populations, 
such  as  low-income  families  who  do  not  participate  in  TANF,  whose 
employment,  earnings,  and  well-being  can  be  compared  with  those  of 
TANF  recipients,  assuming  adequate  sample  sizes  for  both  groups. 
Moreover,  because  NSAF  has  sample  sizes  in  13  states  large  enough  to 
allow  state-level  analyses,  the  employment,  earnings,  and  well-being  of 
TANF  recipients  in  those  states  can  be  considered  in  relation  to  the  state’s 
TANF  programs  and  policies.  However,  using  the  NSAF  data  for  such 
analyses  would  require  resolving  the  challenges  to  analyzing  effects 
described  earlier  in  this  report.  Similarly,  although  most  of  the 
metropolitan  and  community-based  evaluation  samples  do  not  include  pre- 
TANF  welfare  recipients,  other  populations  represented  in  the  study 
samples  could  be  compared  with  TANF  recipients.  Finally,  waiver  and 
demonstration  data  can  be  analyzed  to  gain  information  about  TANF’s 
effects,  keeping  in  mind  that  this  information  is  about  the  effects  of 
programs  and  provisions  often  implemented  prior  to  TANF  and 
implemented  in  cities  and  counties  rather  than  entire  states. 


Concluding 

Observations 


The  data  available  for  addressing  TANF’s  goals  will  provide  useful 
information,  but  with  some  limitations.  Given  the  costs,  some  limitations 
may  be  difficult  to  overcome.  Our  examination  of  the  data  raised  three 
issues. 

First,  for  a  comprehensive  assessment  of  TANF,  it  is  important  to  have  data 
for  a  representative  sample  of  TANF  recipients  and  nonrecipients  that 
allow  for  analyses  of  effect  at  the  state  level.  The  federal  government  has 
made  an  investment  in  national  surveys,  which  either  in  whole  or  in  part 
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are  intended  to  gather  information  about  the  lives  of  TANF  recipients.  One 
of  these,  the  SPD,  was  funded  as  a  means  to  gather  data  about  TANF 
recipients.  For  another,  the  SIPP,  the  Census  Bureau  added  a  special 
section  of  questions  about  welfare  and  reworded  questions  so  that  they 
would  better  capture  respondents’  participation  in  state  programs. 
However,  even  with  these  efforts,  none  of  Census’  surveys  currently  being 
administered  can  be  used  for  state-level  analyses  of  TANF’s  effects  because 
of  small  sample  sizes  within  individual  states.  In  addition,  the  SPD  has  a 
high  attrition  rate.  The  Census  Bureau  plans  to  take  steps  to  improve 
response  to  the  SPD  through  intensive  follow-up  with  survey  dropouts  to 
enlist  their  participation  and  through  monetary  incentives  for  future 
respondents  to  participate  in  the  survey.  However,  the  issue  of  small 
sample  sizes  at  the  state  level  will  remain  unresolved. 

Second,  HHS  has  encouraged  state  agencies  to  study  the  effects  of  their 
TANF  programs  through  the  AFDC  waiver  requirement  for  experimental 
studies  and  subsequent  research  initiatives.  Moreover,  our  examination  of 
data  indicates  that,  because  of  the  variability  in  TANF  program  provisions 
across  states,  analysis  of  TANF’s  effects  at  the  state  and  local  levels  can  be 
done  with  the  greatest  confidence.  However,  even  when  conducted  at  the 
state  and  local  levels,  studies  designed  to  examine  TANF’s  effects  tend  to 
be  costly,  time-consuming,  and  impractical  to  implement  in  every  state.  In 
some  cases,  conducting  an  evaluation  for  an  entire  state  is  determined  to 
be  so  expensive  that  data  collection  is  limited  to  a  portion  of  the  state.  For 
example,  the  evaluation  of  Vermont’s  waiver  program  focused  on  6  of  12 
welfare  service  districts.  The  evaluation’s  42-month  follow-up  survey  was 
administered  to  only  these  6  district  offices  and,  owing  to  cost  constraints, 
included  a  subset  of  the  sample  for  whom  administrative  records,  rather 
than  survey  responses,  were  collected.  Policymakers,  federal  and  state 
officials,  and  the  welfare  reform  research  community  will  need  to  seek 
ways  to  balance  the  need  for  information  about  TANF’s  effects  with  the 
resource  demands  of  rigorous  studies. 

Third,  both  qualitative  and  quantitative  data  may  be  needed  to  understand 
what  has  happened  to  former  TANF  recipients.  Leavers  are  a  difficult 
population  to  track,  and,  in  some  cases,  using  multiple  methods  of 
quantitative  data  collection  has  not  necessarily  increased  the  number  of 
former  recipients  who  could  be  located  or  who  responded  to  surveys.  In 
fact,  in  some  of  the  studies  we  reviewed,  the  low  rate  of  success  in 
gathering  data  from  these  individuals  makes  the  data’s  usefulness 
questionable.  Surveys  that  used  only  one  mode  of  data  collection,  such  as 
telephoning  former  TANF  recipients,  generally  had  the  lowest  response 
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rates.  Some  leavers’  studies  followed  telephone  surveys  with  personal 
interviews  of  those  who  could  not  be  reached  by  phone  or  who  did  not 
respond.  However,  even  the  use  of  multiple  modes  of  data  collection  did 
not  always  ensure  high  response  rates.  Given  the  difficulties  inherent  in 
collecting  quantitative  data  from  this  group,  other  data  collection  strategies 
that  use  local  communication  networks  to  identify  families  as  well  as 
interviews  of  respondents  in  their  homes  may  be  needed  to  gain 
information  about  the  lives  of  TANF  leavers. 


Agency  Comments 


In  commenting  on  a  draft  of  this  report,  HHS  said  that  the  report  will  be  of 
help  to  the  Congress  and  other  interested  parties.  In  its  technical 
comments,  HHS  expressed  concern  that  in  highlighting  the  importance  of 
statewide  samples,  we  understated  the  value  of  data  from  local  samples.  In 
response  to  this  concern,  we  have  noted  in  the  report  not  only  that  findings 
from  local  samples  are  important  but  also  that,  in  some  cases,  they  provide 
data  only  recently  available  from  national  surveys.  We  concur  with  HHS 
that  a  sample  need  not  be  statewide  in  order  for  findings  to  be  useful. 
However,  we  have  emphasized  the  value  of  data  that  can  be  generalized  to 
the  state  level  because  of  the  Congress’  interest  in  a  picture  of  TANF’s 
progress  nationwide.  HHS’  comments  appear  in  appendix  IV. 


As  agreed  with  your  offices,  unless  you  publicly  announce  its  contents 
earlier,  we  plan  no  further  distribution  of  this  report  until  30  days  after  its 
issue  date.  At  that  time,  we  will  send  copies  to  the  Honorable  Tommy  G. 
Thompson,  Secretary  of  Health  and  Human  Services;  appropriate 
congressional  committees;  and  other  interested  parties.  We  also  will  make 
copies  available  to  others  on  request. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me 
on  (202)  512-7202  or  David  D.  Beilis  on  (415)  904-2272.  Another  GAO 
contact  and  staff  acknowledgments  are  listed  in  appendix  V. 

Cynthia  M.  Fagnoni 
Managing  Director 

Education,  Workforce,  and  Income  Security  Issues 
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Appendix  I 


Scope  and  Methodology 


This  appendix  discusses  in  more  detail  our  scope  and  methodology  for 
identifying,  selecting,  and  assessing  studies  and  surveys  that  might  provide 
data  to  help  researchers  as  they  seek  to  describe  what  has  happened  to 
recipients  of  Temporary  Assistance  for  Needy  Families  (TANF)  and  to 
estimate  the  effect  of  welfare  reform  on  them. 


Sample  Design 


Because  no  comprehensive  list  of  data  sources  for  welfare  reform  research 
exists,  we  used  a  judgmental  sampling  method  for  our  assessment  of  data 
resources.  We  began  our  work  by  examining  six  key  critiques  of  welfare 
reform  research  that  had  been  issued,  in  draft  or  final  form,  by  the  fall  of 
1999.  The  six  critiques  listed  in  figure  8  both  gave  us  insight  into  issues  that 
will  probably  arise  in  assessing  TANF  and  identified  studies  that  may  be 
potential  sources  of  data  for  an  assessment  of  TANF.1 


'The  critiques  we  reviewed  are  critical  examinations  of  ongoing  and  planned  research  on 
issues  arising  from  the  welfare  reform  law  (P.L.  104-193).  The  critiques  were  conducted  by 
an  individual  or  representative  of  a  research  organization  known  for  work  on  social  policy 
issues,  and  they  considered  the  methodological  issues  involved  in  designing  an  evaluation  of 
the  welfare  reform  law’s  provisions,  research  issues  that  merit  investigation,  or  both. 
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Figure  8:  Six  Major  Critiques  of  Welfare  Reform  Research 


•  Besharov,  Douglas  J.,  Peter  Germanis,  and  Peter  H.  Rossi. 

Evaluating  Welfare  Reform:  A  Guide  for  Scholars  and  Practitioners. 

College  Park,  Md.:  Welfare  Reform  Academy,  University  of  Maryland  School  of  Public  Affairs,  1997. 

•  Blum,  Barbara  B.,  and  Ellen  C.  Berrey. 

Welfare  Research  Perspectives:  Past,  Present  and  Future. 

New  York,  N.Y.:  Research  Forum  on  Children,  Families  and  the  New  Federalism,  National  Center  for 

Children  in  Poverty,  Columbia  University,  Aug.  1 999. 

•  Midwest  Welfare  Peer  Assistance  Network  (WELPAN). 

Welfare  Reform:  How  Will  We  Know  If  It  Works? 

Washington,  D.C.:  Family  Impact  Seminar,  Jan.  1998. 

•  National  Research  Council,  Committee  on  National  Statistics.  Moffit,  Robert  A.,  and  Michele  Ver  Ploeg,  eds. 
Evaluating  Welfare  Reform:  A  Framework  and  Review  of  Current  Work— Interim  Report. 

Washington,  D.C.:  National  Academy  Press,  Jan.  1999. 

•  Rossi,  Peter  H. 

Research  on  PRWORA  and  TANF:  What  Can  Be  Learned  From  Four  Major  Projects  Currently  Underway. 
Paper  given  at  a  meeting  of  the  Welfare  Reform  Academy,  Washington,  D.C.,  Dec.  1 999. 

•  U.S.  Department  of  Health  and  Human  Services,  Office  of  the  Assistant  Secretary  for  Planning  and 

Evaluation. 

Interim  Status  Report  on  Research  on  the  Outcomes  of  Welfare  Reform. 

Washington,  D.C.:  HHS,  June  1999. 

We  started  the  development  of  a  list  of  data  sources  from  three  of  the 
critiques — the  Research  Forum’s  report  and  its  related  on-line  database, 
the  National  Research  Council’s  interim  report,  and  Peter  Rossi’s  paper.  To 
ensure  that  this  list  was  comprehensive,  we  consulted  with  officials  at  the 
Department  of  Health  and  Human  Services  (HHS)  about  important  bodies 
of  work  in  the  welfare  reform  research  field.  We  also  conducted  follow-up 
interviews  with  HHS  project  officers  and  experts  in  the  welfare  reform 
research  community  to  ensure  that  we  had  identified  the  most  relevant 
national  surveys  and  studies,  particularly  those  that  might  have  data  about 
family,  marriage,  and  pregnancy  issues.  As  a  result  of  these  discussions  and 
an  examination  of  the  original  list,  we  designed  a  judgmental  sample  of 
potential  data  sources  for  welfare  reform  research  that  included  the 
following  categories: 

•  national  surveys  and  HHS’  TANF  administrative  data; 
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•  studies  that  collected  data  about  the  major  TANF  subpopulations  in 
three  or  more  states  or  municipalities; 

•  studies  of  TANF  leavers; 

•  HHS’  waiver  evaluations;  and 

•  studies  listed  on  the  websites  of  HHS’  Administration  for  Children  and 
Families  (ACF),  HHS’  Office  of  the  Assistant  Secretary  for  Planning  and 
Evaluation  (ASPE),  and  the  Welfare  Information  Network  of  the  Finance 
Project.2 

We  then  began  to  develop  lists  of  the  surveys  and  studies  in  each  of  the 
sample’s  categories. 

The  national  surveys  included  in  our  list  were  the  Current  Population 
Survey  (CPS),  the  National  Longitudinal  Survey  of  Youth  (NLSY),  the 
National  Survey  of  America’s  Families  (NSAF),  the  Panel  Study  of  Income 
Dynamics  (PSID),  the  Survey  of  Income  and  Program  Participation  (SIPP), 
and  the  Survey  of  Program  Dynamics  (SPD).  We  used  information  from 
ASPE  and  from  the  National  Conference  of  State  Legislatures  to  identify 
leavers  studies  sponsored  by  HHS  or  states.  Similarly,  we  used  information 
from  ACF  to  ensure  that  our  list  contained  the  body  of  research  funded  by 
ACF  that  focused  on  waivers  implemented  by  state  welfare  agencies  prior 
to  TANF’s  authorization.  As  we  added  items  to  the  list,  we  continually 
checked  to  avoid  any  duplication.  This  comparison  involved  our  judgment, 
as  some  lists  were  of  projects  or  studies  and  others  were  of  study  reports. 
Because  we  relied  on  multiple  reviews  of  the  body  of  work  undertaken  in 
the  welfare  reform  research  community,  we  believe  that  the  list  of  443 
entries  we  compiled  included  the  key  sources  of  data. 


Sample  Selection 


We  selected  surveys  and  studies  systematically  from  this  list  within  each 
sample  category.  We  were  interested  in  surveys  or  studies  that  were  as 
comprehensive  as  possible  in  geographic  coverage  and  topics  addressed. 
Thus,  we  selected  all  of  the  national  surveys  and  the  HHS  administrative 
data.  We  also  selected  all  studies  on  the  original  list  that  by  their 
description  appeared  to  have  produced  data  concerning  the  major 
subpopulations  affected  by  TANF  in  three  or  more  states,  municipalities,  or 
counties.  This  resulted  in  55  studies  and  surveys. 


Ahe  Finance  Project  is  a  nonprofit  research,  technical  assistance,  and  information 
organization  created  to  improve  outcomes  for  children,  families,  and  communities 
nationwide. 
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We  then  selected  studies  that  pertained  to  individual  states  in  the  following 
way.  First  we  selected  all  leavers  studies  financed  by  ASPE.  Of  the  leavers 
studies  listed  by  the  National  Conference  of  State  Legislatures  and  those 
mentioned  in  an  article  authored  by  Brauner  and  Loprest,3  we  included 
only  those  that  had  not  been  included  in  our  previous  report4  or  were  not 
from  a  state  that  already  had  an  ASPE-funded  study.  In  states  that  had 
issued  multiple  reports  for  their  leavers  studies  for  people  who  left  welfare 
in  different  years,  we  selected  the  most  recent  study.  When  a  state  had  no 
ASPE-funded  study  or  any  listed  by  the  National  Conference  of  State 
Legislatures  or  Brauner  and  Loprest,  but  did  have  a  report  available  on  its 
Web  site,  we  selected  the  Web  report.  Waiver  studies  generally  produced 
several  reports.  We  selected  for  review  the  most  recently  issued  waiver 
report  because  the  data  topics  examined  were  similar  in  the  initial  and  later 
reports. 

After  selecting  these  types  of  studies  and  surveys,  we  removed  from  our  list 
studies  that  did  not  appear  to  contain  data  that  could  answer  our  research 
questions  or  that  used  data  from  one  of  the  national  surveys  on  our  list.  In 
summary,  we  excluded  literature  searches,  reviews  of  research  on  state 
policies  or  programs,  technical  assistance  projects  focused  on  improving 
or  evaluating  information  systems  or  databases,  and  studies  based  on  data 
from  a  national  survey  that  we  had  included  in  our  list.  A  list  of  239  studies 
remained.  Finally,  we  obtained  advice  from  five  welfare  experts  about 
which  of  these  239  studies  we  should  include.  Ultimately,  we  selected  17  of 
these  studies. 

In  all,  we  judgmentally  selected  141  national  surveys  and  studies  that 
yielded  187  data  sets  to  review.  A  complete  list  of  the  national  surveys  and 
studies  that  we  examined  for  data  is  provided  in  appendix  II. 


Data  Assessment  Identifying  data  resources  for  a  comprehensive  assessment  of  TANF 

required  criteria  that  could  be  used  to  assess  data  sets.  The  first  step  in  this 
process  was  to  express  each  of  TANF’s  goals  as  a  research  question.  In 
looking  at  the  goals  themselves,  it  is  evident  that  some  express  expected 
results — for  example,  that  work  and  marriage  will  improve  the  well-being 


3Brauner,  Sarah,  and  Pamela  Loprest,  Where  Are  They  Now?  What  States’  Studies  of  People 
Who  Left  Welfare  Tell  Us  (Washington,  D.C.:  The  Urban  Institute,  May  1999.) 

4GAO/HEHS-99^8,  Apr.  28,  1999. 
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of  low-income  families.  Assessing  TANF’s  progress  toward  these  expected 
results  required,  in  part,  questions  about  TANF’s  effects.  However,  some  of 
TANF’s  goals  focus  on  its  general  purpose — for  example,  providing 
assistance  to  needy  families.  In  this  case,  assessing  TANF’s  progress 
required  research  questions  that  are  descriptive,  that  is,  questions  that  ask 
what  public  assistance  looks  like  under  TANF.  To  translate  TANF’s  goals 
into  research  questions,  we  considered  the  nature  of  each  of  TANF’s  goals 
and  formulated  questions  to  represent  key  issues  the  Congress  will 
consider  at  reauthorization.  As  shown  in  figure  1,  we  created 
corresponding  questions  that  asked  for  descriptions  of  what  has  happened 
under  TANF,  the  effects  of  TANF,  or  both. 

We  then  specified  the  information,  or  data  topics,  necessary  to  address  our 
research  questions.  We  developed  a  data  collection  instrument  that  listed 
the  data  topics  associated  with  each  question  and  used  the  instrument  to 
record  the  data  topics  found  in  each  data  set  examined.  It  is  important  to 
note  that  what  we  identified  as  data  topics  were  not  equivalent  to  specific 
measures.  In  other  words,  our  coding  captured  the  fact  that  a  certain  data 
source  collected  measures  on  employment.  It  did  not  capture  the  specific 
manner  in  which  employment  was  measured.  In  addition  to  data  topics,  we 
collected  such  pertinent  information  as  the  unit  of  analysis,  population, 
sampling  method,  sample  size,  dates  covered  by  data  collection,  and  design 
of  the  study  for  which  data  were  gathered.  We  recorded  response  rates  and 
attrition  rates  when  they  were  relevant  given  the  method  of  data  collection. 
We  also  looked  to  see  if  data  had  been  or  were  being  collected  for  a 
comparison  or  control  group. 

To  summarize  our  findings,  we  identified  data  categories  related  to  TANF’s 
goals,  some  of  which  represented  the  research  questions  and  others  of 
which  were  more  narrowly  focused.  The  narrowly  focused  data  categories 
represented  combinations  of  data  topics,  such  as  employment  and  earnings 
or  family  and  child  well-being,  that  were  associated  with  the  research 
questions.  We  took  this  approach  for  a  variety  of  reasons.  First,  in  making  a 
judgment  that  data  were  available  to  address  particular  questions,  we 
required  that  certain  data  topics  be  present  in  combination  and,  for  effect 
questions,  that  the  data  were  collected  using  control  groups  or  comparison 
groups.  However,  a  data  source  could  provide  relevant  data  topics,  even 
though  the  data  topic  could  not  be  used  to  address  the  particular  question 
we  had  posed.  Rather  than  discount  the  value  of  these  data  topics,  we 
decided  to  note  their  availability.  Second,  in  many  cases,  the  same  data 
topics  and  data  categories  were  being  used  to  address  different  questions. 
For  example,  as  figure  1  shows,  the  data  categories  associated  with 
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employment  were  related  to  5  of  our  11  questions.  Presenting  our  findings 
in  terms  of  data  categories  allowed  us  to  report  on  all  of  the  data  topics, 
including  those  that  were  not  available  in  the  combinations  needed  to 
address  a  research  question. 

Finally,  to  assess  how  the  data  might  be  used  for  an  assessment  of  TANF, 
we  considered  three  attributes  of  the  data.  We  considered  the  geographic 
scope  of  the  sample;  the  data  topics  included  in  the  data  set;  and  whether 
or  not  the  data  could  be  used  for  descriptive  analyses  or  analyses  of  effect, 
given  the  design  of  the  study.  In  determining  the  geographic  scope  of  the 
sample,  we  looked  at  the  sampling  method  and  sample  size,  as  well  as  at 
response  rates  and  attrition  rates,  since  both  affect  how  well  a  sample 
represents  a  population.  We  relied  on  the  design  of  the  study,  the  data 
topics  included  in  a  data  set,  and  how  researchers  had  used  the  data  to 
make  a  judgment  about  whether  the  data  could  be  used  for  descriptive 
analyses  or  analyses  of  effect.  We  coded  data  as  being  useable  for  analyses 
of  effect  when  they  came  from  a  study  that  made  comparisons  between 
groups,  one  of  which  served  as  the  treatment  group  and  the  other  as  the 
absence  of  the  treatment,  or  the  comparison  group.  In  deciding  whether  a 
study  included  a  treatment  and  a  comparison  group,  we  recognized  that 
such  groups  could  be  formed  through  experimental  designs,  quasi- 
experimental  designs,  or  statistical  modeling. 


Study  Limitations 


Because  this  assessment  is  based  on  a  judgmental  sample  and  the  data 
needs  of  an  assessment  of  TANF’s  progress  are  derived  from  TANF’s 
legislative  objectives,  several  study  limitations  should  be  considered.  First, 
while  every  attempt  was  made  to  be  comprehensive  in  sample  design  and 
selection,  some  relevant  data  sources  may  have  been  omitted.  Second, 
framing  the  data  needs  for  an  assessment  of  TANF’s  progress  around 
TANF’s  objectives,  which  focus  on  the  behavior  and  well-being  of  low- 
income  children  and  families,  excluded  from  consideration  the  bodies  of 
welfare  reform  research  concerned  with  institutions,  including  studies  of 
TANF’s  implementation  at  the  state  and  local  levels  and  descriptions  of 
TANF  program  policies  and  practices.  Third,  the  study’s  focus  on 
identification  of  quantitative  data  resulted  in  our  eliminating  data  from 
most  studies  that  used  qualitative  data  collection  methods.  Fourth,  because 
our  bibliographic  sources  for  surveys  and  studies  included  both  existing 
and  planned  surveys  and  studies,  complete  documentation  for  data  sets 
was  not  always  available.  Finally,  because  our  coding  focused  on  whether  a 
certain  data  source  collected  measures  on  specific  topics,  but  not  on  the 
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precise  measures  used,  we  did  not  assess  whether  measures  were 
comparable  across  studies. 
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Surveys  and  Welfare  Reform  Studies 
Reviewed 


Table  5:  National  Data  Sources 

Source 

Sponsor 

HHS  administrative  data 

HHS 

CPS 

Census  Bureau 

NLSY 

Bureau  of  Labor  Statistics,  Department  of  Labor 

NSAF 

The  Urban  Institute 

PSID 

University  of  Michigan,  Institute  for  Social 

Research 

SIPP 

Census  Bureau 

SPD 

Census  Bureau 

Table  6:  Sources  of  Waiver,  Demonstration,  Leavers,  Metropolitan  and  Community-Based,  and  Other  Welfare  Reform  Data 

Study  title 

Principal  investigator 

Scope 

Data  source 

Location 

Waiver  data 

Evaluation  of  the  Arizona  EMPOWER 

Welfare  Reform  Demonstration:  Impact 

Study,  Interim  Report 

Abt  Associates 

Local 

Administrative  and 
survey 

Three  sites  in  Phoenix 
and  one  site  on  a  Navajo 
reservation,  Arizona 

Jobs  First:  Implementation  and  Early  Impacts 
of  Connecticut’s  Welfare  Reform  Initiative 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative  and 
survey 

New  Haven  and 

Manchester,  Connecticut 

The  Family  Transition  Program: 
Implementation  and  Three-Year  Impacts  of 
Florida’s  Initial  Time  Limited  Welfare 

Program 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative  and 
survey 

Escambia  County,  Florida 

Indiana  Welfare  Reform  Evaluation:  Program 
Implementation  and  Economic  Impacts  After 
Two  Years 

Abt  Associates,  The  Urban 
Institute 

State 

Administrative  and 
survey 

Indiana 

Iowa’s  Family  Investment  Program:  Impacts 
During  the  First  Three-and-a-Half  Years  of 
Welfare  Reform 

Mathematica  Policy 
Research 

Local 

Administrative 

Nine  counties  in  Iowa — 
five  urban  and  four  rural 

Iowa’s  Limited  Benefit  Plan 

Mathematica  Policy 
Research 

State 

Administrative 

Iowa 

Iowa’s  Limited  Benefit  Plan 

Mathematica  Policy 
Research 

Local 

Survey  and  case 
study 

Three  PROMISE  JOBS 
service  delivery  areas, 

Iowa 

Second  Assignments  to  Iowa’s  Limited 

Benefit  Plan 

Mathematica  Policy 
Research 

State 

Administrative, 
survey,  and  case 
study 

Iowa 

Reforming  Welfare  and  Rewarding  Work: 

Final  Report  on  the  Minnesota  Family 
Investment  Program 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative  and 
two  surveys 

Three  urban  and  four 
rural  counties  in 

Minnesota 
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(Continued  From  Previous  Page) 

Study  title 

Principal  investigator 

Scope 

Data  source 

Location 

Achieving  Change  for  Texans  Evaluation:  Net 
Impacts  Through  December  1997 

University  of  Texas  at 
Austin,  Center  for  the 

Study  of  Human 

Resources 

Local 

Administrative 

10  counties  in  Texas 

Recipients  of  the  Texas  One-Time  Benefit 
Payment:  An  Interim  Report 

University  of  Texas  at 
Austin,  Center  for  Social 
Work  Research 

Local 

Survey 

Hidalgo  and  Cameron 
counties,  Texas 

Forty-Two-Month  Impacts  of  Vermont’s 
Welfare  Restructuring  Project 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative  and 
survey 

6  of  12  welfare  district 
offices,  Vermont 

Wisconsin  Self-Sufficiency  First/Pay  for 
Performance  Program:  Results  and  Lessons 
From  a  Social  Experiment 

University  of  Wisconsin- 
Madison,  Institute  for 
Research  on  Poverty 

Local 

Administrative 

Dane,  Dodge,  Jefferson, 
and  Waukesha  counties 
in  Wisconsin 

Year  Three  Progress  Report:  Customer 
Characteristics  and  Employment  Patterns 

University  of  Maryland, 
Baltimore,  School  of  Social 
Work 

State 

Administrative 

Maryland 

Interim  Report  for  the  Enhancement  to  the 
Process  and  Impact  Analysis  of  the  Youth 
Employment  and  Training  Initiative 

Illinois  State  University, 
Department  of  Social  Work 

Local 

Administrative  and 
survey 

Chicago,  Illinois 

Montana  FAIM  Evaluation:  Evaluation 

Design3 

Abt  Associates 

State 

Administrative  and 
survey 

Montana 

A  Proposal  to  Complete  the  Evaluation  of 
Employment  First,  Nebraska’s  Welfare 

Reform  Program3 

Mathematica  Policy 
Research 

State 

Survey 

Nebraska 

Evaluation  of  Welfare  Reform  in  New 
Hampshire:  Mid-Evaluation  Summary  Report 

University  of  New 
Hampshire 

State 

Administrative  and 
two  surveys 

New  Hampshire 

Evaluation  of  the  North  Carolina  Work  First 
Program:  Initial  Analysis  of  Administrative 
Data 

Maximus 

State 

Administrative 

North  Carolina 

Evaluation  of  North  Dakota’s  Training, 
Education,  Employment,  and  Management 
Program:  Final  Report 

Berkeley  Planning 
Associates 

Local 

Administrative 

Seven  counties  in  North 
Dakota 

Early  Impacts  of  the  Virginia  Independence 
Program:  Final  Report 

Mathematica  Policy 
Research 

Local 

Administrative 

Lynchburg,  Petersburg, 
and  Portsmouth;  and 

Prince  William  and  Wise 
counties,  Virginia 

An  Overview  and  Synthesis  of  the  Project  of 
State  Level  Child  Outcomes 

Child  Trends 

State  and 
local 

Survey 

Indiana;  New  Haven  and 
Manchester,  Connecticut; 
Alachua  and  Escambia 
counties,  Florida;  nine 
counties  in  Iowa;  and 
seven  counties  in 

Minnesota 
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(Continued  From  Previous  Page) 

Study  title 

Principal  investigator 

Scope 

Data  source 

Location 

Final  Impact  Report:  The  Evaluation  of  “To 
Strengthen  Michigan  Families” 

Abt  Associates 

Local 

Administrative 

Four  local  service  offices: 
Kalamazoo  (Kalamazoo 
County),  Madison  Heights 
(Oakland  County), 
McNichols/Goddard,  and 
Schaeffer/Six  Mile 
(Wayne  County), 

Michigan 

A  Final  Report  on  the  Impact  of  New  Jersey’s 
Family  Development  Program 

Rutgers  University 

Local 

Administrative 

10  counties  in  New 

Jersey 

A  Final  Report  on  the  Impact  of  New  Jersey’s 
Family  Development  Program:  Results  From 
a  Pre-Post  Analysis  of  AFDC  Case  Heads 
From  1990-1996 

Rutgers  University 

State 

Administrative 

New  Jersey 

Ohio  Works  First  Evaluation  Plan3 

Macro  International 

State 

Administrative  and 
survey 

Ohio 

California  Welfare  Reform  Impact  Evaluation 

University  of  California, 
Berkeley 

b 

b 

b 

Findings  From  Year  Two  of  the  Minnesota 
Work  First — MFIP  Evaluation 

Maximus 

b 

b 

b 

Process  Study  of  South  Carolina  Family 
Independence  Program 

Urban  Institute 

b 

b 

b 

Demonstration  data 

Struggle  to  Sustain  Employment:  The 
Effectiveness  of  the  Post-Employment 
Services  Demonstration 

Mathematica  Policy 
Research 

Local 

Administrative  and 
survey 

Chicago,  Illinois; 

Riverside,  California; 
Portland,  Oregon;  and 

San  Antonio,  Texas 

National  Evaluation  of  Welfare-to-Work 
Strategies:  Evaluating  Alternative  Welfare-to- 
Work  Approaches:  Two-Year  Impacts  for 
Eleven  Programs 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative  and 
survey 

Atlanta,  Georgia; 

Columbus,  Ohio;  Detroit 
and  Grand  Rapids, 
Michigan;  Oklahoma  City, 
Oklahoma;  Portland, 
Oregon;  and  Riverside, 
California 

National  Evaluation  of  Welfare-to-Work 
Strategies:  Impacts  on  Young  Children  and 
Their  Families  Two  Years  After  Enrollment: 
Findings  From  the  Child  Outcomes  Study 

Manpower  Demonstration 
Research  Corporation 

Local 

Survey 

Atlanta,  Georgia;  Grand 
Rapids,  Michigan;  and 
Riverside,  California 

New  Hope  for  People  With  Low  Incomes: 
Two-Year  Results  of  a  Program  to  Reduce 
Poverty  and  Reform  Welfare 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative 

Milwaukee,  Wisconsin 

The  New  York  State  Child  Assistance 
Program:  Five-Year  Impacts,  Costs  and 
Benefits 

Abt  Associates 

Local 

Administrative 

Niagara,  Monroe,  and 
Suffolk  counties,  New 

York 

Building  Opportunities,  Enforcing 

Obligations:  Implementation  and  Interim 
Impacts  of  Parents’  Fair  Share 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative 

Dayton,  Ohio;  Grand 

Rapids,  Michigan;  and 
Memphis,  Tennessee 
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Study  title 

Principal  investigator 

Scope 

Data  source 

Location 

Teenage  Parent  Demonstration 

Mathematica  Policy 
Research 

Local 

Administrative  and 
survey 

Camden  and  Newark, 

New  Jersey;  Chicago, 

Illinois 

Blueprints  for  Violence  Prevention:  Prenatal 
and  Infancy  Home  Visitation  by  Nurses 

University  of  Colorado, 
Cornell  University, 

University  of  Rochester, 
and  University  of  Denver 

Local 

Administrative  and 
survey 

Memphis,  Tennessee; 
Elmira,  New  York 

New  Chance:  Final  Report  on  a 
Comprehensive  Program  for  Young  Mothers 
in  Poverty  and  Their  Children 

Manpower  Demonstration 
Research  Corporation 

Local 

Survey 

16  locations  in  California, 
Colorado,  Florida,  Illinois, 
Kentucky,  Michigan, 
Minnesota,  New  York, 
Oregon,  and 

Pennsylvania 

The  National  Evaluation  of  the  Welfare-to- 
Work  Grants  Program 

Mathematica  Policy 
Research,  The  Urban 
Institute,  and  Support 
Services  International 

Local 

c 

8  to  10  localities 
nationwide 

Leavers  data 

Evaluation  of  the  North  Carolina  Work  First 
Program:  Status  of  Families  Leaving  Work 
First  After  Reaching  the  24-Month  Time  Limit 

Maximus 

State 

Survey 

North  Carolina 

Arizona  Cash  Assistance  Exit  Study:  First 
Quarter  1998  Cohort  Final  Report 

Arizona  Department  of 
Economic  Security,  Office 
of  Evaluation 

State 

Administrative  and 
survey 

Arizona 

Arkansas  Well-Being  Survey 

Berkeley  Policy  Associates 

State 

Survey 

Arkansas 

Los  Angeles  County  Post-TANF  Tracking 
Study 

Manpower  Demonstration 
Research  Corporation 

Local 

Survey 

Los  Angeles  County, 
California 

Bay  Area  Family  Well-Being  Survey:  A  Study 
of  Welfare  Leavers  in  Alameda  and  Contra 
Costa  Counties 

The  Sphere  Institute 

Local 

Administrative  and 
survey 

Contra  Costa  and 

Alameda  counties, 

California 

Examining  the  Circumstances  of  Individuals 
and  Families  Who  Leave  TANF:  Assessing 
the  Validity  of  Administrative  Data 

The  Sphere  Institute 

Local 

Administrative  and 
survey 

San  Mateo  County, 
California 

Evaluation  of  the  Colorado  Works  Program: 
First  Annual  Report  1999 

Berkeley  Planning 
Associates 

State 

Administrative  and 
survey 

Colorado 

The  Status  of  TANF  Leavers  in  the  District  of 
Columbia:  Interim  Report 

The  Urban  Institute 

State 

Administrative  and 
survey 

District  of  Columbia 

After  Leaving  WAGES 

Florida  State  University 

State 

Survey 

Florida 

Georgia  Welfare  Leavers  Study:  Initial 

Results 

Georgia  State  University, 
Applied  Research  Center 

State 

Survey 

Georgia 

When  Families  Leave  Welfare  Behind:  First 
Survey  Findings  for  Illinois  Families  in 
Transition 

University  of  Illinois  at 
Springfield,  Institute  for 
Public  Affairs,  and 

University  of  Illinois  at 
Urbana-Champaign, 

School  of  Social  Work 

State 

Administrative  and 
survey 

Illinois 
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Study  title 

Principal  investigator 

Scope 

Data  source 

Location 

A  Survey  of  Kansas  Households  Leaving  the 
Temporary  Assistance  for  Families  Program: 
Final  Report 

Kansas  Department  of 
Social  and  Rehabilitation 
Services 

State 

Administrative  and 
survey 

Kansas 

How  Are  They  Doing?  A  Longitudinal  Study 
Tracking  Households  Leaving  Welfare  Linder 
Massachusetts’  Reform 

Massachusetts 

Department  of  Transitional 
Assistance 

State 

Survey 

Massachusetts 

Tracking  of  TANF  Clients,  First  Report  of  a 
Longitudinal  Study  (Mississippi’s  TANF 
program) 

Millsaps  College,  Center 
for  Applied  Research 

Local 

Survey 

Eight  counties  in 

Mississippi 

Preliminary  Outcomes  for  1996  Fourth 
Quarter  AFDC  Leavers:  Revised  Interim 
Report 

Missouri  Department  of 
Social  Services  and 
University  of  Missouri, 
Department  of  Economics 

State 

Administrative 

Missouri 

Missouri  Leavers  Project:  Three  Chapter 
Report 

Midwest  Research  Institute 

State 

Survey 

Missouri 

After  Welfare:  A  Study  of  Work  and  Benefit 
Use  in  New  York  State  After  Case  Closing, 
December  1999 

State  University  of  New 
York,  Rockefeller  Institute 
of  Government 

State 

Administrative 

New  York 

Employment  and  Return  to  Public  Assistance 
Among  Single  Female-Headed  Families 
Leaving  AFDC  in  Third  Quarter  1996, 
Cuyahoga  County,  Ohio 

Case  Western  Reserve 
University,  Center  on 

Urban  Poverty  and  Social 
Change,  and  Manpower 
Demonstration  Research 
Corporation 

Local 

Administrative 

Cuyahoga  County,  Ohio 

How  Are  They  Managing?  A  Six-Month 
Retrospective  of  Cuyahoga  County  Families 
Leaving  Welfare:  Fourth  Quarter  1998  and 
First  Quarter  1999 

Case  Western  Reserve 
University,  Center  on 

Urban  Poverty  and  Social 
Change 

Local 

Survey 

Cuyahoga  County,  Ohio 

Survey  of  Former  Family  Independence 
Program  Clients:  Cases  Closed  During  April 
Through  September  1998 

South  Carolina 

Department  of  Social 
Services 

State 

Survey 

South  Carolina 

Texas  Families  in  Transition:  The  Impacts  of 
Welfare  Reform  Changes  in  Texas:  Early 
Findings 

Texas  Department  of 
Human  Services 

State 

Survey 

Texas 

The  Virginia  Closed  Case  Study: 

Experiences  of  Virginia  Families  One  Year 
After  Leaving  Temporary  Assistance  for 
Needy  Families 

Virginia  Polytechnic 

Institute  and  State 

University  and 

Mathematica  Policy 
Research 

State 

Administrative  and 
survey 

Virginia 

Experiences  of  Virginia  Time  Limit  Families 
in  the  Six  Months  After  Case  Closure: 

Results  for  Early  Cohort:  Final  Report, 
November  1999 

Mathematica  Policy 
Research 

Local 

Administrative  and 
survey 

Virginia  districts  2,6,7, 
and  9 
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Study  title 

Principal  investigator 

Scope 

Data  source 

Location 

A  Study  of  Washington  State  TANF  Leavers 
and  TANF  Recipients:  Findings  From  the 
April-June  1999  Telephone  Survey:  Final 
Report 

Washington  Department  of 
Social  and  Health 

Services,  Economic 
Services  Administration, 
Office  of  Planning  and 
Research 

State 

Administrative  and 
survey 

Washington 

A  Study  of  Washington  State  TANF 
Departures  and  Welfare  Reform:  TANF  Exit 
Study  April  1999 

Washington  Department  of 
Social  and  Health 

Services,  Economic 
Services  Administration, 
Management  Reports  and 
Data  Analysis 

State 

Administrative 

Washington 

Before  and  After  TANF:  The  Economic  Well- 
Being  of  Women  Leaving  Welfare 

University  of  Wisconsin- 
Madison,  Institute  for 
Research  on  Poverty 

State 

Administrative 

Wisconsin 

Wisconsin  Works  (W-2):  First  Quarter  1998 
Leavers  Study:  Preliminary  Process  Report 

Wisconsin  Department  of 
Workforce  Development 

State 

Survey 

Wisconsin 

Connecticut’s  Post-Time-Limit  Tracking 

Study:  Six-Month  Survey  Results 

Manpower  Demonstration 
Research  Corporation 

Local 

Survey 

Department  of  Social 
Services  offices  in 
Bridgeport,  Hartford, 
Manchester,  New  Haven, 
Norwich,  and  Waterbury, 
Connecticut 

CalWORKS  Leavers  Survey:  A  Statewide 
Telephone  Survey  of  Former  CalWORKS 
Recipients 

California  Department  of 
Social  Services 

State 

Survey 

California 

Metropolitan  and  community-based  data 

The  Fragile  Families  and  Child  Well-Beinq 
Study 

Princeton  University  and 
Columbia  University 

Local 

Adminstrative 
and  survey 

Austin,  Texas;  Baltimore, 
Maryland;  Detroit, 

Michigan;  Newark,  New 
Jersey;  Oakland, 

California;  Philadelphia, 
Pennsylvania;  and 
Richmond,  Virginia 

Los  Angeles  Families  and  Neighborhood 
Survey  (LA  FANS) 

RAND 

Local 

Survey 

Los  Angeles 
neighborhoods 

Welfare  Children  and  Families:  A  Three  City 
Study 

Brandeis  University, 
University  of  Chicago, 
University  of  Illinois, 
University  of  North 

Carolina  at  Chapel  Hill, 
University  of  Texas  at 
Austin,  Johns  Hopkins 
University,  Pennsylvania 
State  University,  and 
Harvard  University 

Local 

Survey 

Boston,  Massachusetts; 

San  Antonio,  Texas;  and 
Chicago,  Illinois 
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Study  title 

Principal  investigator 

Scope 

Data  source 

Location 

Growing  Up  in  Poverty  Project:  Remember 
the  Children:  Mothers  Balance  Work  and 
Child  Care  Under  Welfare  Reform 

University  of  California- 
Berkeley  and  Yale 
University 

Local 

Survey 

San  Francisco  and  Santa 
Clara,  California;  Tampa, 
Florida;  and  New  Haven, 
Connecticut 

Big  Cities  and  Welfare  Reform:  Early 
Implementation  and  Ethnographic  Findings 
From  the  Project  on  Devolution  and  Urban 
Change 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative  and 
survey 

One  county  each  in  Ohio, 
California,  Florida,  and 
Pennsylvania 

Impact  of  Welfare  Reform  on  Families 

University  of  Wisconsin- 
Madison,  Institute  for 
Research  on  Poverty 

Local 

Survey 

Milwaukee,  Wisconsin 

National  Institute  of  Child  Health  and 
Development  Study  of  Early  Child  Care 

National  Institute  of  Child 
Health  and  Development 

Local 

Ethnographic  and 
survey 

In  or  near  Little  Rock, 
Arkansas;  Morganton, 

North  Carolina;  Seattle, 
Washington;  Madison, 
Wisconsin;  Philadelphia 
and  Pittsburgh, 
Pennsylvania;  Boston, 
Massachusetts; 

Lawrence,  Kansas;  Irvine, 
California;  and 
Charlottesville,  Virginia 

Other  welfare  reform  data 

Work  First  New  Jersey  (WFNJ)  Evaluation: 
Flow  WFNJ  Clients  Are  Faring  Under  Welfare 
Reform:  An  Early  Look 

Mathematica  Policy 
Research 

State 

Administrative  and 
survey 

New  Jersey 

Barriers  to  the  Employment  of  Welfare 
Recipients 

University  of  Michigan, 
School  of  Social  Work 

Local 

Survey 

One  urban  Michigan 
county 

Florida’s  Project  Independence:  Benefits, 
Costs,  and  Two-Year  Impacts  of  Florida’s 
JOBS  Program 

Manpower  Demonstration 
Research  Corporation 

Local 

Administrative  and 
survey 

Nine  counties  in  Florida 

aThis  document  is  an  evaluation  design  plan. 


bWe  were  unable  to  obtain  a  copy  of  either  the  report  or  a  design  plan. 
cThis  information  was  missing  from  the  documents  we  reviewed. 
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Table  7:  Other  Studies  Examined 

Study  or  project  title 

Principal  investigator  or  author 

What  Has  Welfare  Accomplished? 

Impacts  on  Welfare  Participation, 
Employment,  Income,  Poverty,  and  Family 
Structure 

Robert  F.  Schoeni  and  Rebecca  M.  Blank, 
National  Bureau  of  Economic  Research 

National  Study  of  Child  Care  for  Low- 
Income  Families 

Abt  Associates 

Children  and  Welfare  Reform:  A  Guide  to 
Evaluating  the  Effects  of  State  Welfare 
Policies  on  Children 

Child  Trends 

Patterns  and  Growth  of  Child  Care 

Voucher  Use  by  Families  Connected  to 

Cash  Assistance  in  Illinois  and  Maryland 

National  Center  for  Children  in  Poverty, 
Columbia  University 

National  Survey  of  Child  and  Adolescent 
Well-Being 

Research  Triangle  Institute 

The  Role  of  Intermediaries  in  Linking 

TANF  Recipients  With  Jobs 

Mathematica  Policy  Research 

Rural  Welfare-to-Work  Strategies: 

Research  Synthesis 

Macro  International 

Mecklenburg  County  Linked  Database 

Mecklenburg  County  Department  of 

Social  Services,  South  Carolina 

South  Carolina  Child  Link 

South  Carolina  Department  of  Social 
Services 

Policy  Evaluation  of  the  Overall  Effects  of 
Welfare  Reform  on  SSA  Programs 

The  Lewin  Group 

Temporary  Assistance  for  Low-Wage 
Workers:  Evolving  Relationships  Among 
Work,  Welfare,  and  Unemployment 
Insurance 

National  Governors’  Association 

Welfare  Time  Limits:  An  Interim  Report 

Card 

Manpower  Demonstration  Research 
Corporation 

Parenting  Behavior  in  a  Sample  of  Young 
Mothers  in  Poverty:  Results  of  the  New 
Chance  Observational  Study 

Manpower  Demonstration  Research 
Corporation 

Approaches  to  Evaluating  Welfare 

Reform:  Lessons  From  Five  State 
Demonstrations 

Mathematica  Policy  Research 

Expanding  Health  Insurance  Coverage  for 
Low-Income  People:  Experiments  in  Five 
States 

Mathematica  Policy  Research 

Helping  Young  People  in  High-Poverty 
Communities:  Lessons  From  Youth  Fair 
Chance 

Mathematica  Policy  Research 
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Surveys  and  Welfare  Reform  Studies 
Reviewed 


(Continued  From  Previous  Page) 


Study  or  project  title 

Principal  investigator  or  author 

Working  Out  of  Poverty:  Employment 
Retention  and  Career  Advancement  for 
Welfare  Recipients 

National  Governors’  Association 

Welfare  Reform  in  California:  State  and 
County  Implementation  of  CALWORKS  in 
the  First  Year 

RAND 

Welfare  Reform  in  California:  Results  of 
the  1998  All-County  Implementation 

Study 

RAND 

State  Capacity  Study 

Field  Research  Report  Form  1997 

State  University  of  New  York,  The  Nelson 

A.  Rockefeller  Institute  of  Government 

Building  Bridges:  States  Respond  to 
Substance  Abuse  and  Welfare  Reform 

National  Center  on  Addiction  and 

Substance  Abuse,  Columbia  University 

Big  Cities  Confront  the  New  Politics  of 

Child  and  Family  Policy 

Columbia  University  School  of  Social 

Work 

A  Description  and  Assessment  of  State 
Approaches  to  Diversion  Programs  and 
Activities  Under  Welfare  Reform 

Center  for  Health  Policy  Research, 

George  Washington  University 

Diversion  as  a  Work-Oriented  Welfare 
Reform  Strategy  and  Its  Effect  on  Access 
to  Medicaid:  An  Examination  of 

Experiences  of  Five  Local  Communities 

Center  for  Health  Policy  Research, 

George  Washington  University 

An  Unfair  Head  Start:  California  Families 
Face  Gaps  in  Preschool  and  Child  Care 
Availability 

Berkeley-Stanford  Pace  Center,  Yale 
University,  and  California  Child  Care 
Resource  and  Referral  Network 

The  Infant  Health  and  Development 
Program:  Interim  Summary 

IHDP  Research  Group 

National  Impact  Evaluation  of  the 
Comprehensive  Child  Development 
Program:  Final  Report 

Abt  Associates 

Illegal  Aliens:  Extent  of  Welfare  Benefits 
Received  on  Behalf  of  U.S.  Citizen 

Children 

U.S.  General  Accounting  Office 

Implementing  Welfare  Reform 

Requirements  for  Teenage  Parents: 

Lessons  From  Experience  in  Four  States 

Mathematica  Policy  Research 

Medicaid  Managed  Care:  Does  It 

Increase  Primary  Care  Services  in 
Underserved  Areas? 

Mathematica  Policy  Research 

New  Immigrant  Survey 

RAND 

Welfare  Reform  Project 

Ohio  University  and  Joyce  Foundation 

Study  of  the  Employment  Patterns  of 

Young  Women  and  the  Implications  for 
Welfare  Mothers 

The  Urban  Institute 
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Surveys  and  Welfare  Reform  Studies 
Reviewed 


(Continued  From  Previous  Page) 


Study  or  project  title 

Principal  investigator  or  author 

Alameda  County  CalWORKS  Needs 
Assessment:  A  Look  at  Potential  Health- 
Related  Barriers  to  Self-Sufficiency 

Public  Health  Institute 

The  Study  of  Families  Formerly  Receiving 
Aid  to  Families  With  Dependent  Children, 
Interim  Report:  1999  Survey 

Midwest  Research  Institute 

New  Mexico  TANF  Longitudinal  Study 
Results  of  First- Year  Follow-up  Surveys: 
Final  Report 

Maximus 

Welfare  Reform  After  Two  Years: 

Technical  Report  on  Former  Welfare 
Recipients  in  Pennsylvania 

Pennsylvania  Department  of  Public 

Welfare 

Indiana  Welfare  Reform  Evaluation: 
Identifying  and  Serving  the  Most 

Dependent  Cases 

Abt  Associates 

A  Study  of  Washington  State  TANF 

Leavers  and  TANF  Recipients:  Welfare 
Reform  and  Findings  From  Administrative 
Data:  Final  Report 

Washington  Department  of  Social  and 
Health  Services 

Post-Exit  Earnings  and  Benefit  Receipt 
Among  Those  Who  Left  AFDC  in 

Wisconsin 

University  of  Wisconsin-Madison,  Institute 
for  Research  on  Poverty 

Evaluation  of  the  Arizona  EMPOWER 
Welfare  Reform  Demonstration:  Process 
Study,  Interim  Report 

Abt  Associates 

Work  and  Welfare:  Iowa  Families  Tell 

Their  Stories 

Mathematica  Policy  Research 

Process  Evaluation  of  Achieving  Change 
for  Texans:  Welfare  Reform  Waiver 
Evaluation,  First  Interim  Report 

Texas  Department  of  Human  Services 

Will  Welfare  Reform  Influence  Marriage 
and  Fertility?  Early  Evidence  From  the 

ABC  Demonstration 

Abt  Associates 

The  Next  Generation:  The  Effects  of 

Welfare  Reform  and  Employment  Policies 
on  Children  and  Families 

Manpower  Demonstration  Research 
Corporation 

The  Effect  of  Welfare  on  Marriage  and 
Fertility 

Robert  Moffitt  in  Welfare,  the  Family,  and 
Reproductive  Behavior:  Research 
Perspectives 

New  Research  Findings  on  the  Effects  of 
the  Earned  Income  Tax  Credit 

Center  on  Budget  and  Policy  Priorities 

When  Low-Income  Mothers  Go  to  Work: 
Implications  for  Children 

Child  Trends 

Work,  Earnings,  and  Well-Being  After 
Welfare:  What  Do  We  Know? 

University  of  Wisconsin-Madison,  Institute 
for  Research  on  Poverty 
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Surveys  and  Welfare  Reform  Studies 
Reviewed 


(Continued  From  Previous  Page) 


Study  or  project  title 

Principal  investigator  or  author 

Monitoring  the  Impact  of  Welfare  Reform 
on  Women’s  Health 

Johns  Hopkins  University  Women’s  and 
Children’s  Health  Policy  Center 

Monitoring  the  Impact  of  Welfare  Reform 
on  Immigrant  Women,  Infants,  and 

Children:  Access  to  Health  Care,  Health- 
Seeking  Behaviors,  and  Health  Outcomes 

New  York  City  Department  of  Health, 

Bureau  of  Maternity  Services  and  Family 
Planning 

Building  an  Employment-Focused  Welfare 
System:  Work  First  and  Other  Work- 
Oriented  Strategies  in  Five  States 

The  Urban  Institute 

Welfare  Waivers  and  Non-Marital  Child 
Bearing 

Ann  Horvath  and  H.  Elizabeth  Peters, 
Cornell  University 

Marriage  and  Economic  Incentives: 

Evidence  From  a  Welfare  Experiment 

Wei-Yin  Hu,  University  of  California  at  Los 
Angeles 

Welfare  Benefits  and  Female  Headship  in 
the  United  States 

Robert  Moffitt  in  American  Economic 
Review,  Vol.  90,  No.  2  (2000),  pp.  373-77. 

Note:  These  studies  were  examined  for  data  and  were  found  to  include  (1)  data  that  could  not  be  used 
to  answer  our  research  questions,  (2)  data  that  represented  a  secondary  analysis,  or  (3)  data  that 
duplicated  what  was  found  in  other  studies  reviewed. 
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Scope  and  Type  of  Analyses  for  Which 
and  Demonstration  Data  Can  Be  Used 

Waiver 

Study  or  project  title 

Descriptive  analyses 

Analyses  of 
effect 

Statewide  sample 

Indiana  Welfare  Reform  Evaluation:  Program  Implementation  and  Economic 

Impacts  After  Two  Years 

•  Administrative 

•  Survey 

X 

Iowa’s  Limited  Benefit  Plan 
•  Administrative 

X 

Second  Assignments  to  Iowa’s  Limited  Benefit  Plan 

•  Administrative 

•  Survey 

•  Case  study 

X 

Year  Three  Progress  Report:  Customer  Characteristics  and  Employment  Patterns 
•  Administrative 

X 

Montana  FAIM  Evaluation:  Evaluation  Design 

•  Administrative 

•  Survey 

X 

A  Proposal  to  Complete  the  Evaluation  of  Employment  First,  Nebraska’s  Welfare 
Reform  Program 
•  Survey 

X 

Evaluation  of  Welfare  Reform  in  New  Hampshire:  Mid-Evaluation  Summary  Report 

•  Administrative 

•  Surveys  (2) 

X 

Evaluation  of  the  North  Carolina  Work  First  Program:  Initial  Analysis  of 

Administrative  Data 
•  Administrative 

X 

An  Overview  and  Synthesis  of  the  Project  on  State-Level  Child  Outcomes 
•  Survey 

X 

A  Final  Report  on  the  Impact  of  New  Jersey’s  Family  Development  Program: 

Results  From  a  Pre-Post  Analysis  of  AFDC  Case  Heads  From  1990  to  1996 
•  Administrative 

X 

Ohio  Works  First  Evaluation  Plan 

•  Administrative 

•  Survey 

X 

Local  sample 

Evaluation  of  Arizona  EMPOWER  Welfare  Reform  Demonstration:  Impact  Study, 
Interim  Report 

•  Administrative 

•  Survey 

X 

Jobs  First:  Implementation  and  Early  Impacts  of  Connecticut’s  Welfare  Reform 

Initiative 

•  Administrative 

•  Survey 

X 
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Scope  and  Type  of  Analyses  for  Which  Waiver 

and  Demonstration  Data  Can  Be  Used 

(Continued  From  Previous  Page) 

Analyses  of 

Study  or  project  title 

Descriptive  analyses 

effect 

The  Family  Transition  Program:  Implementation  and  Three-Year  Impact  of  Florida’s 

Initial  Time-Limited  Welfare  Program 

•  Administrative 

•  Survey 

X 

Iowa’s  Family  Investment  Program:  Impacts  During  the  First  3-y2  Years  of  Welfare 

Reform 

X 

•  Administrative 

Iowa’s  Limited  Benefit  Plan 

X 

•  Survey 

•  Case  study 

Reforming  Welfare  and  Rewarding  Work:  Final  Report  on  the  Minnesota  Family 
Investment  Program 

•  Administrative 

•  Surveys  (2) 

X 

Achieving  Change  for  Texans  Evaluation:  Net  Impacts  Through  December  1997 
•  Administrative 

X 

Recipients  of  the  Texas  One-Time  Benefit  Payment:  An  Interim  Report 
•  Survey 

X 

Forty-Two-Month  Impacts  of  Vermont’s  Welfare  Restructuring  Project 

•  Administrative 

•  Survey 

X 

Wisconsin  Self-Sufficiency  First/Pay  for  Performance  Program:  Results  and 

Lessons  From  a  Social  Experiment 
•  Administrative 

X 

Interim  Report  for  the  Enhancement  to  the  Process  and  Impact  Analysis  of  the 

Youth  Employment  and  Training  Initiative 

•  Administrative 

•  Survey 

X 

Evaluation  of  North  Dakota’s  Training,  Education,  Employment,  and  Management 
Program:  Final  Report 
•  Administrative 

X 

Early  Impacts  of  the  Virginia  Independence  Program:  Final  Report 
•  Administrative 

X 

Final  Impact  Report:  The  Evaluation  of  “To  Strengthen  Michigan  Families” 

•  Administrative 

X 

A  Final  Report  on  the  Impact  of  New  Jersey’s  Family  Development  Program 
•  Administrative 

X 

Struggle  to  Sustain  Employment:  The  Effectiveness  of  the  Post-Employment 

Services  Demonstration 

X 

•  Administrative 

•  Survey 

National  Evaluation  of  Welfare-to-Work  Strategies:  Evaluating  Alternative  Welfare- 
to-Work  Approaches:  Two-Year  Impacts  for  Eleven  Programs 

•  Administrative 

•  Survey 

X 
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Scope  and  Type  of  Analyses  for  Which  Waiver 
and  Demonstration  Data  Can  Be  Used 


(Continued  From  Previous  Page) 

Analyses  of 

Study  or  project  title 

Descriptive  analyses  effect 

National  Evaluation  of  Welfare-to-Work  Strategies:  Impacts  on  Young  Children  and 
Their  Families  Two  Years  After  Enrollment:  Findings  From  the  Child  Outcomes 

Study 
•  Survey 

X 

New  Hope  for  People  With  Low  Incomes:  Two-Year  Results  of  a  Program  to 

Reduce  Poverty  and  Reform  Welfare 
•  Administrative 

X 

The  New  York  State  Child  Assistance  Program:  Five-Year  Impacts,  Costs,  and 

Benefits 
•  Administrative 

X 

Building  Opportunities,  Enforcing  Obligations:  Implementation  &  Interim  Impacts  of 
Parents’  Fair  Share 
•  Administrative 

X 

Teenage  Parent  Demonstration 
•  Administrative 

X 

•  Survey 

Blueprints  for  Violence  Prevention:  Prenatal  and  Infancy  Home  Visitation  by  Nurses 

•  Administrative 

•  Survey 

X 

New  Chance:  Final  Report  on  a  Comprehensive  Program  for  Young  Mothers  in 

Poverty  and  Their  Children 
•  Survey 

X 

The  National  Evaluation  of  the  Welfare-to-Work  Grants  Program 
•  Survey 

X 

An  Overview  and  Synthesis  of  the  Project  on  State-Level  Child  Outcomes 
•  Survey 

X 
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Comments  From  the  Department  of  Health 
and  Human  Services 


DEPARTMENT  OF  HEALTH  &  HUMAN  SERVICES 


ADMINISTRATION  FOR  CHILDREN  AND  FAMILIES 

Office  of  the  Assistant  Secretary,  Suite  600 
370  L'Enfant  Promenade,  S.W. 

Washington,  D.C.  20447 


Janaury  29,  2001 


Ms.  Cynthia  M.  Fagnoni 
Managing  Director 

Education,  Workforce,  and  Income  Security  Issues 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 

Dear  Ms  Fagnoni: 

Thank  you  for  the  opportunity  to  comment  on  the  General  Accounting  Office  draft  report 
entitled:  Welfare  Reform:  Data  Available  to  Assess  TANF's  Progress  (GAQ-01-298).  Our 
comments,  which  include  those  from  the  Office  of  the  Assistant  Secretary  for  Planning  and 
Evaluation,  are  enclosed. 

This  report  should  prove  very  useful  in  helping  Congress  and  other  interested  parties  get  a  better 
framework  and  understanding  of  the  opportunities  and  limitations  of  using  available  data  to 
inform  decisions  regarding  reauthorization  of  the  TANF  program.  If  you  have  any  questions 
regarding  our  comments,  please  contact  Karl  Koerper  from  the  Office  of  Planning,  Research  and 
Evaluation  at  (202)  401-4535. 


Sincerely, 


Diann  Dawson 

Acting  Principal  Deputy  Assistant  Secretary 
for  Children  and  Families 


Enclosure 
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GAO  Contacts  and  Staff  Acknowledgments 


GAO  Contacts  David  D.  Beilis,  (415)  904-2272 

Sara  E.  Edmondson,  (202)  512-8516 


Staff 

Acknowledgments 


In  addition  to  those  named  above,  the  following  individuals  made 
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Larin,  and  Heather  McCallum  provided  consultation  on  TANF  policy  and 
implementation  issues. 
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